It is incredibly unlikely to find yourself in the world where the significant insights about real doomsday is coming from single visionary who did so little that can be unambiguously graded, before coming up with those insights.
I think this is a mistaken picture of the intellectual history around AI risk.
Prominent AI folk like Hans Moravec and Marvin Minsky had predicted the eclipse of humanity and humane values (save perhaps as pets/specimens/similar, losing the overwhelming majority of the future) long before Yudkowsky. Moravec in particular published a fair bit of his analysis in his books Mind Children and Robot, including a prediction of the eventual “devouring” of humanity by competitive AI life (stored as data and perhaps occasionally pulled out as simulations). Many other AI researchers endorse this rough picture, although often approvingly, saying that these would be “worthy successors” and “humans aren’t worth saving” or “nothing to be done about it” and so forth.
Vinge’s (a mathematician as well as a sci-fi novelist) 1993 essay has the phrase “How to survive in the post-human era” in the title, and “If not to be avoided, can events be guided so that we may survive?” and discusses risks to humanity extensively (as large).
I.J. Good mentioned the risk of out-of-control AI in the first public writing on the concept of an intelligence explosion.
Marcus Hutter, Jurgen Schmidhuber, Kevin Warwick, and a number of other AI folk have written about the future of AI and risk of human extinction, etc.
Stephen Omohundro is another successful AI guy who has done work in this area.
The concept of “Friendly AI,” i.e. an AI with a clear enough model of human welfare to fairly reliably design successors that get better and better at helping humans, was originally created by the philosopher Nick Bostrom, not Yudkowsky.
Yudkowsky has spent more time on the topic than any of the others on this list, and has specific conclusions that are more idiosyncratic (especially the combination of views on many subjects), but the basic ideas are not so rare or privileged that they do not recur independently among many folk, including subject matter experts.
It appears to me that will to form most accurate beliefs about the real world, and implement solutions in the real world, is orthogonal to problem solving itself.
Problem solving works better when you can flexibly reallocate internal resources to old and new uses that will best meet goal criteria, run experiments (at least internal ones, involving writing and using programs), and identify and pursue subtasks.
Why would optimizing compiler that can optimize it’s ability to optimize, suddenly emerge will?
If it considers, creates, and runs programs in the course of identifying and evaluating possible improvements (with respect to some criteria of improvement), then it doesn’t need to acquire some novel will. What is the necessary difference between creating and running a program that does a specialized search or simulation on local hardware and returns an answer, and a program that transmits an AI into the outside world to acquire resources and return an answer? Even more so, if the AI is designed to communicate with outside sources of information like humans or experimental apparatus and use them as “black-boxes” to acquire information.
It is immediately conjectured...That is rationalization. Privileging a path of thought.
If you make a point, and someone raises a complication that undercuts your conclusion, it may look to you like it is an instant rationalization to your novel objection. But in fact the points you raise (simulation-based and heuristic instrumental reasons for AI to be wary of immediately killing humans, wireheading, etc) and the counter-considerations you identify as rationalizations, are old news to many of your interlocutors (people like Wei Dai), and were independently invented (by supposed doomsayers) in the course of searching the possibility space, for both good and bad news. ETA: See Wei Dai’s link below.
This is not to say that there aren’t rationalization and biases of discourse on this subject around here: there are comments and commenters that clearly illustrate those.
Prominent AI folk like Hans Moravec and Marvin Minsky had predicted the eclipse of humanity and humane values (save perhaps as pets/specimens/similar, losing the overwhelming majority of the future) long before Yudkowsky.
A minor point maybe, but...how big is the fraction of all AI researchers and computer scientists who fall into that category?
I.J. Good, Marcus Hutter, Jurgen Schmidhuber, Kevin Warwick, Stephen Omohundro, Vinge etc.
Those are really a handful of names. And their practically useful accomplishments are little. Most AI researcher would consider them dreamers.
Yudkowsky has spent more time on the topic than any of the others on this list,
This is frequently mentioned but bears little evidence. Many smart people like Roger Penrose spent a lot of time on their pet theories. That does not validate them. It just allowed them to find better ways to rationalize their ideas.
I.J. Good, Marcus Hutter, Jurgen Schmidhuber, Kevin Warwick, Stephen Omohundro, Vinge etc.
Those are really a handful of names. And their practically useful accomplishments are little. Most AI researcher would consider them dreamers.
Good was involved in the very early end of computers, so it is a bit hard for him to have done modern AI work. But the work he did do was pretty impressive. He did cryptography work in World War II with Alan Turing, and both during and after the war worked on both theoretical and practical computer systems. He did a lot of probability work, much of which is in some form or another used today in a variety of fields including AI. For example, look at the Good-Turing estimator.
Schmidhuber did some of the first work on practical genetic algorithms and did very important work on neural nets.
Warwick has done so much work in AI and robotics that listing them all would take a long time. One can argue that most of it hasn’t gone outside the lab, but it is clear that the much of that work is practically useful even if it is not yet economically feasible to use it on a large scale (which frankly is the status of most AI research at this point in general).
Overall, I don’t think your characterization is accurate, although your point that the total set of AI researchers with such concerns being a small percentage of all researchers seems valid.
Prominent AI folk like Hans Moravec and Marvin Minsky had predicted the eclipse of humanity and humane values (save perhaps as pets/specimens/similar, losing the overwhelming majority of the future) long before Yudkowsky.
A minor point maybe, but...how big is the fraction of all AI researchers and computer scientists who fall into that category?
I.J. Good, Marcus Hutter, Jurgen Schmidhuber, Kevin Warwick, Stephen Omohundro, Vinge etc.
A strange list, IMO. Fredkin was one who definitely entertained the “pets” hypothesis:
It was rumoured in some of the UK national press of the time that Margaret Thatcher watched Professor Fredkin being interviewed on a late night TV science programme. Fredkin explained that superintelligent machines were destined to surpass the human race in intelligence quite soon, and that if we were lucky they might find human beings interesting enough to keep us around as pets.
It was generated by selecting (some) people who had written publications in the area, not merely oral statements. Broadening to include the latter would catch many more folk.
I was thinking of “Robot: from mere machine to transcendent mind” where he talks about an era in which humans survive through local tame robots, but eventually are devoured by competitive minds that have escaped beyond immediate control.
The most impressive on your list (e.g. Good) also are the earliest; in particular ‘intelligence explosion’ predates computational complexity theory which puts severe bounds on any foom scenarios.
Of the later people with important contributions, I’m not even sure why you have Hutter on the list; I guess misrepresenting Hutter is some local tradition that I didn’t pick up on back then. When you are vague with regards to what was said, it is difficult to verify you, which I guess is how this work. But if you keep doing this eventually you’re going to piss off someone with 10x the notability of S.I.
And none of that is relevant—it is incredibly improbable that the world saving organisation would look that incompetent.
The most impressive on your list (e.g. Good) also are the earliest; in particular ‘intelligence explosion’ predates computational complexity theory which puts severe bounds on any foom scenarios.
I think there is a trend to this effect (although Solomonoff wrote about intelligence explosion in 1985). I wouldn’t point to computational complexity though, so much as general disappointment in AI progress.
How do you think I am misrepresenting Hutter? I agree that he is less influential than Good, and not one of the best-known names in AI. If you are talking about his views on possible AI outcomes, I was thinking of passages like the one in this Hutter paper:
Let us now consider outward explosion, where an increasing amount of
matter is transformed into computers of fixed efficiency (fixed comp per unit
time/space/energy). Outsiders will soon get into resource competition with the
expanding computer world, and being inferior to the virtual intelligences, probably
only have the option to flee. This might work for a while, but soon the expansion
rate of the virtual world should become so large, theoretically only bounded by the
speed of light, that escape becomes impossible, ending or converting the outsiders’
existence.
So while an inward explosion is interesting, an outward explosion will be a threat
to outsiders. In both cases, outsiders will observe a speedup of cognitive processes
and possibly an increase of intelligence up to a certain point. In neither case will
outsiders be able to witness a true intelligence singularity.
I think there is a trend to this effect. I wouldn’t point to computational complexity though, so much as general disappointment in AI progress.
Well, the self improvement would seem a lot more interesting if it was the case that P=NP or P=PSPACE , I’d say. As it is a lot of scary things are really well bounded—e.g. specific, accurate prediction of various nonlinear systems requires exponential knowledge, exponential space, and exponential number of operation, in a given forecast time. And the progress is so disappointing perhaps thanks to P!=NP and the like—the tasks do not have easy general solutions, or even general purpose heuristics.
re: quote
Ahh, that’s much better with regard to vagueness. He isn’t exactly in agreement with SI doctrine, though, and the original passage creates impression of support for the specific doctrine here.
It goes to say that optimistic AI researchers consider AI to be risky, which is definitely a good thing for the world but at the same time makes this rhetoric in the vibe of ‘other AI researchers are going to kill everyone, and we are the only hope of humanity’ look rather bad. The researchers that aren’t particularly afraid of AI seem to be working on fairly harmless projects which just aren’t coding for that sort of will to paperclip.
Suppose some group says that any practical nuclear reactor will intrinsically risk a multimegaton nuclear explosion. What could that really mean? One thing really: the approach that they consider practical will intrinsically risk a multimegaton nuclear explosion. It doesn’t say much about other designs, especially if that group doesn’t have a lot of relevant experience. Same ought to apply to SI’s claims.
‘other AI researchers are going to kill everyone, and we are the only hope of humanity’
Let me explicitly reject such rhetoric then.
The difficulty of safety is uncertain: it could be very easy for anyone with little time, or it could be quite difficult and demand a lot of extra work (which might be hard to put in given competitive pressures). The region where safety depends sensitively on the precautions and setup of early AI development (from realistic options) should not be much larger than the “easy for everyone region,” so trivially the probability for building AI with good outcomes should be distributed widely among the many possible AI building institutions: software firms, government, academia, etc. And since a small team is very unlikely to build AGI first, it can have at most only a very small share of the total expected probability of a good outcome.
A closed project aiming to build safe AI could have an advantage either by using more demanding safety thresholds and by the possibility of not publishing results that require additional work to make safe but could be immediately used for harm. This is the reasoning for classifying some kinds of work with dangerous viruses or nuclear technology or the like. This could provide some safety boost for such a project in principle, but probably not an overwhelming one.
Secrecy might also be obtained through ordinary corporate and government security, and governments in particular would plausibly be much better at it (the Manhattan Project leaked, but ENIGMA did not). And different safety thresholds matter most with respect to small risks (most institutions would be worried about large risks, whereas those more concerned with future generations might place extra weight on small risks). But small risks contribute less to expected value.
And I would very strongly reject the idea that “generic project X poses near-certain doom if it succeeds while project Y is almost certain to have good effects if it succeeds”: there’s just no way one could have such confident knowledge.
And the progress is so disappointing perhaps thanks to P!=NP and the like—the tasks do not have easy general solutions, or even general purpose heuristics.
You can still get huge differences in performance from software. Chess search explodes as you go deeper, but software improvements have delivered gains comparable to hardware gains: the early AI people were right that if they had been much smarter they could have designed a chess program to beat the human world champion using the hardware they had.
Part of this is that in chess one is interested in being better than one’s opponent: sure you can’t search perfectly 50 moves ahead, but you don’t have to play against an infinite-computing-power brute-force search, you have to play against humans and other computer programs. Finance, computer security, many aspects of military affairs, and other adversarial domains are pretty important. If you could predict the weather a few days further than others, you could make a fortune trading commodities and derivatives.
Another element is that humans are far from optimized to use their computation for chess-playing, which is likely true for many of the other activities of modern civilization.
Also, there’s empirical evidence from history and firm R&D investments that human research suffers from serial speed limits of human minds, i.e. one gets more progress from doubling time to work than the size of the workforce. This is most true in areas like mathematics, cryptography, and computer science, less true in areas demanding physical infrastructure built using the outputs of many fields and physically rate-limited processes. But if one can rush forward on those elements, there would then be an unprecedented surge of ability to advance the more reluctant physical technologies.
I mentioned how vague it is; it is impossible for anyone to check what is exactly meant without going over literally everything Hutter ever wrote.
Hutter was much less ambiguously misrepresented/misquoted during the more recent debate with Holden Karnofsky (due to the latter’s interest in AIXI), so I am assuming, by the process of induction, that same happened here.
it is impossible for anyone to check what is exactly meant without going over literally everything Hutter ever wrote.
As it happens, I looked it up and did this ‘impossible’ task in a few seconds before I replied, because I expected the basis for your claim to be as lame as it is; here’s the third hit in Google for ‘marcus hutter ai risk’: “Artificial Intelligence: Overview”
Slide 67 includes some of the more conventional worries like technological unemployment and abuse of AI tools; more importantly, slide 68 includes a perfectly standard statement of Singularity risks, citing, as it happens, Moravec, Goode, Vinge, and Kurzweil; I’ll quote it in full (emphasis added):
What If We Do Succeed?
The success of AI might mean the end of the human race.
Artificial evolution is replaced by natural solution. AI systems will be our mind children (Moravec 2000)
Once a machine surpasses the intelligence of a human it can design even smarter machines (I.J.Good 1965).
This will lead to an intelligence explosion and a technological singularity at which the human era ends.
Prediction beyond this event horizon will be impossible (Vernor Vinge 1993)
Alternative 1: We keep the machines under control.
Alternative 2: Humans merge with or extend their brain by AI. Transhumanism (Ray Kurzweil 2000)
Let’s go back to what Carl said:
Marcus Hutter, Jurgen Schmidhuber, Kevin Warwick, and a number of other AI folk have written about the future of AI and risk of human extinction, etc.
Sure sounds like ‘Marcus Hutter...have written about the future of AI and risk of human extinction’.
Which in that case demonstrates awareness among the AI researchers of the risk, while at the same time not demonstrating that Hutter finds it particularly likely that this would happen (‘might’) or agrees with any specific alarmist rhetoric. I can’t know if that’s what Carl actually refers to. I do assure you that about every AI researcher has seen the Terminator.
I gave the Hutter quote I was thinking of upthread.
My aim was basically to distinguish between buying Eliezer’s claims and taking intelligence explosion and AI risk seriously, and to reject the idea that the ideas in question came out of nowhere. One can think AI risk is worth investigating without thinking much of Eliezer’s views or SI.
I agree that the cited authors would assign much lower odds of catastrophe given human-level AI than Eliezer. The same statement would be true of myself, or of most people at SI and FHI: Eliezer is at the far right tail on those views. Likewise for the probability that a small team assembled in the near future could build safe AGI first, but otherwise catastrophe would have ensued.
Well, I guess that’s fair enough. In the quote on the top, though, I am specifically criticizing the extreme view. At the end of the day, the entire raison d’etre for SI’s existence is the claim that without paying you the risk would be higher. The claim that you are somehow fairy unique. And there are many risks—for example, risk of lethal flu-like pandemic—which are much more clearly understood and where specific efforts have much more clearly predictable outcome of reducing the risk. Favouring a group of AI theorists but not other does not have clearly predictable outcome of reducing the risk.
(I am inclined to believe that the pandemic is under funded as it would primarily decimate the poorer countries, ending existence of entire cultures, whereas the ‘existential risk’ is a fancy phrase for a risk to the privileged)
Which in that case demonstrates awareness among the AI researchers of the risk, while at the same time not demonstrating that Hutter finds it particularly likely that this would happen (‘might’) or agrees with any specific alarmist rhetoric.
It need not demonstrate any such thing to fit Carl’s statement perfectly and give the lie to your claim that he was misrepresenting Hutter.
I do assure you that about every AI researcher has seen the Terminator.
Sure, hence the Hutter citation of “(Cameron 1984)”. Oh wait.
Yudkowsky has spent more time on the topic than any of the others on this list, and has specific conclusions that are more idiosyncratic (especially the combination of views on many subjects), but the basic ideas are not so rare or privileged that they do not recur independently among many folk, including subject matter experts.
The argument is for the insights coming out of EY , and the privileging that EY is making for those hypotheses originated by others, aka cherrypicking what to advertise. EY is a good writer.
edit: concrete thought example: There is a drug A that undergoes many tests, with some of them evaluating it as better than placebo, some as equal to placebo, and some as worse to placebo. Worst of all, each trial is conducted on 1 person’s opinion. Comes in the charismatic pharmaceutical marketer, or charismatic anti-vaccination campaign leader, and starts bringing to attention the negative or positive trials. That is not good. Even if there’s both of those people.
I think this is a mistaken picture of the intellectual history around AI risk.
Prominent AI folk like Hans Moravec and Marvin Minsky had predicted the eclipse of humanity and humane values (save perhaps as pets/specimens/similar, losing the overwhelming majority of the future) long before Yudkowsky. Moravec in particular published a fair bit of his analysis in his books Mind Children and Robot, including a prediction of the eventual “devouring” of humanity by competitive AI life (stored as data and perhaps occasionally pulled out as simulations). Many other AI researchers endorse this rough picture, although often approvingly, saying that these would be “worthy successors” and “humans aren’t worth saving” or “nothing to be done about it” and so forth.
Vinge’s (a mathematician as well as a sci-fi novelist) 1993 essay has the phrase “How to survive in the post-human era” in the title, and “If not to be avoided, can events be guided so that we may survive?” and discusses risks to humanity extensively (as large).
I.J. Good mentioned the risk of out-of-control AI in the first public writing on the concept of an intelligence explosion.
Marcus Hutter, Jurgen Schmidhuber, Kevin Warwick, and a number of other AI folk have written about the future of AI and risk of human extinction, etc.
Stephen Omohundro is another successful AI guy who has done work in this area.
The concept of “Friendly AI,” i.e. an AI with a clear enough model of human welfare to fairly reliably design successors that get better and better at helping humans, was originally created by the philosopher Nick Bostrom, not Yudkowsky.
Yudkowsky has spent more time on the topic than any of the others on this list, and has specific conclusions that are more idiosyncratic (especially the combination of views on many subjects), but the basic ideas are not so rare or privileged that they do not recur independently among many folk, including subject matter experts.
Problem solving works better when you can flexibly reallocate internal resources to old and new uses that will best meet goal criteria, run experiments (at least internal ones, involving writing and using programs), and identify and pursue subtasks.
If it considers, creates, and runs programs in the course of identifying and evaluating possible improvements (with respect to some criteria of improvement), then it doesn’t need to acquire some novel will. What is the necessary difference between creating and running a program that does a specialized search or simulation on local hardware and returns an answer, and a program that transmits an AI into the outside world to acquire resources and return an answer? Even more so, if the AI is designed to communicate with outside sources of information like humans or experimental apparatus and use them as “black-boxes” to acquire information.
If you make a point, and someone raises a complication that undercuts your conclusion, it may look to you like it is an instant rationalization to your novel objection. But in fact the points you raise (simulation-based and heuristic instrumental reasons for AI to be wary of immediately killing humans, wireheading, etc) and the counter-considerations you identify as rationalizations, are old news to many of your interlocutors (people like Wei Dai), and were independently invented (by supposed doomsayers) in the course of searching the possibility space, for both good and bad news. ETA: See Wei Dai’s link below.
This is not to say that there aren’t rationalization and biases of discourse on this subject around here: there are comments and commenters that clearly illustrate those.
A minor point maybe, but...how big is the fraction of all AI researchers and computer scientists who fall into that category?
I.J. Good, Marcus Hutter, Jurgen Schmidhuber, Kevin Warwick, Stephen Omohundro, Vinge etc.
Those are really a handful of names. And their practically useful accomplishments are little. Most AI researcher would consider them dreamers.
This is frequently mentioned but bears little evidence. Many smart people like Roger Penrose spent a lot of time on their pet theories. That does not validate them. It just allowed them to find better ways to rationalize their ideas.
Good was involved in the very early end of computers, so it is a bit hard for him to have done modern AI work. But the work he did do was pretty impressive. He did cryptography work in World War II with Alan Turing, and both during and after the war worked on both theoretical and practical computer systems. He did a lot of probability work, much of which is in some form or another used today in a variety of fields including AI. For example, look at the Good-Turing estimator.
Schmidhuber did some of the first work on practical genetic algorithms and did very important work on neural nets.
Warwick has done so much work in AI and robotics that listing them all would take a long time. One can argue that most of it hasn’t gone outside the lab, but it is clear that the much of that work is practically useful even if it is not yet economically feasible to use it on a large scale (which frankly is the status of most AI research at this point in general).
Overall, I don’t think your characterization is accurate, although your point that the total set of AI researchers with such concerns being a small percentage of all researchers seems valid.
A strange list, IMO. Fredkin was one who definitely entertained the “pets” hypothesis:
http://www.dai.ed.ac.uk/homes/cam/Robots_Wont_Rule2.shtml#Bad
It was generated by selecting (some) people who had written publications in the area, not merely oral statements. Broadening to include the latter would catch many more folk.
Your list of Hans Moravec and Marvin Minsky was fine—though I believe Moravec characterised humans being eaten by robots as follows:
...though he did go on to say thay he was “not too bothered” by that because “in the long run, that’s how it’s going to be anyway”.
I was more complaining about XiXiDu’s “reframing” of the list.
I was thinking of “Robot: from mere machine to transcendent mind” where he talks about an era in which humans survive through local tame robots, but eventually are devoured by competitive minds that have escaped beyond immediate control.
The most impressive on your list (e.g. Good) also are the earliest; in particular ‘intelligence explosion’ predates computational complexity theory which puts severe bounds on any foom scenarios.
Of the later people with important contributions, I’m not even sure why you have Hutter on the list; I guess misrepresenting Hutter is some local tradition that I didn’t pick up on back then. When you are vague with regards to what was said, it is difficult to verify you, which I guess is how this work. But if you keep doing this eventually you’re going to piss off someone with 10x the notability of S.I.
And none of that is relevant—it is incredibly improbable that the world saving organisation would look that incompetent.
I think there is a trend to this effect (although Solomonoff wrote about intelligence explosion in 1985). I wouldn’t point to computational complexity though, so much as general disappointment in AI progress.
How do you think I am misrepresenting Hutter? I agree that he is less influential than Good, and not one of the best-known names in AI. If you are talking about his views on possible AI outcomes, I was thinking of passages like the one in this Hutter paper:
Well, the self improvement would seem a lot more interesting if it was the case that P=NP or P=PSPACE , I’d say. As it is a lot of scary things are really well bounded—e.g. specific, accurate prediction of various nonlinear systems requires exponential knowledge, exponential space, and exponential number of operation, in a given forecast time. And the progress is so disappointing perhaps thanks to P!=NP and the like—the tasks do not have easy general solutions, or even general purpose heuristics.
re: quote Ahh, that’s much better with regard to vagueness. He isn’t exactly in agreement with SI doctrine, though, and the original passage creates impression of support for the specific doctrine here.
It goes to say that optimistic AI researchers consider AI to be risky, which is definitely a good thing for the world but at the same time makes this rhetoric in the vibe of ‘other AI researchers are going to kill everyone, and we are the only hope of humanity’ look rather bad. The researchers that aren’t particularly afraid of AI seem to be working on fairly harmless projects which just aren’t coding for that sort of will to paperclip.
Suppose some group says that any practical nuclear reactor will intrinsically risk a multimegaton nuclear explosion. What could that really mean? One thing really: the approach that they consider practical will intrinsically risk a multimegaton nuclear explosion. It doesn’t say much about other designs, especially if that group doesn’t have a lot of relevant experience. Same ought to apply to SI’s claims.
Let me explicitly reject such rhetoric then.
The difficulty of safety is uncertain: it could be very easy for anyone with little time, or it could be quite difficult and demand a lot of extra work (which might be hard to put in given competitive pressures). The region where safety depends sensitively on the precautions and setup of early AI development (from realistic options) should not be much larger than the “easy for everyone region,” so trivially the probability for building AI with good outcomes should be distributed widely among the many possible AI building institutions: software firms, government, academia, etc. And since a small team is very unlikely to build AGI first, it can have at most only a very small share of the total expected probability of a good outcome.
A closed project aiming to build safe AI could have an advantage either by using more demanding safety thresholds and by the possibility of not publishing results that require additional work to make safe but could be immediately used for harm. This is the reasoning for classifying some kinds of work with dangerous viruses or nuclear technology or the like. This could provide some safety boost for such a project in principle, but probably not an overwhelming one.
Secrecy might also be obtained through ordinary corporate and government security, and governments in particular would plausibly be much better at it (the Manhattan Project leaked, but ENIGMA did not). And different safety thresholds matter most with respect to small risks (most institutions would be worried about large risks, whereas those more concerned with future generations might place extra weight on small risks). But small risks contribute less to expected value.
And I would very strongly reject the idea that “generic project X poses near-certain doom if it succeeds while project Y is almost certain to have good effects if it succeeds”: there’s just no way one could have such confident knowledge.
You can still get huge differences in performance from software. Chess search explodes as you go deeper, but software improvements have delivered gains comparable to hardware gains: the early AI people were right that if they had been much smarter they could have designed a chess program to beat the human world champion using the hardware they had.
Part of this is that in chess one is interested in being better than one’s opponent: sure you can’t search perfectly 50 moves ahead, but you don’t have to play against an infinite-computing-power brute-force search, you have to play against humans and other computer programs. Finance, computer security, many aspects of military affairs, and other adversarial domains are pretty important. If you could predict the weather a few days further than others, you could make a fortune trading commodities and derivatives.
Another element is that humans are far from optimized to use their computation for chess-playing, which is likely true for many of the other activities of modern civilization.
Also, there’s empirical evidence from history and firm R&D investments that human research suffers from serial speed limits of human minds, i.e. one gets more progress from doubling time to work than the size of the workforce. This is most true in areas like mathematics, cryptography, and computer science, less true in areas demanding physical infrastructure built using the outputs of many fields and physically rate-limited processes. But if one can rush forward on those elements, there would then be an unprecedented surge of ability to advance the more reluctant physical technologies.
How is Hutter being misrepresented here?
I mentioned how vague it is; it is impossible for anyone to check what is exactly meant without going over literally everything Hutter ever wrote.
Hutter was much less ambiguously misrepresented/misquoted during the more recent debate with Holden Karnofsky (due to the latter’s interest in AIXI), so I am assuming, by the process of induction, that same happened here.
As it happens, I looked it up and did this ‘impossible’ task in a few seconds before I replied, because I expected the basis for your claim to be as lame as it is; here’s the third hit in Google for ‘marcus hutter ai risk’: “Artificial Intelligence: Overview”
Slide 67 includes some of the more conventional worries like technological unemployment and abuse of AI tools; more importantly, slide 68 includes a perfectly standard statement of Singularity risks, citing, as it happens, Moravec, Goode, Vinge, and Kurzweil; I’ll quote it in full (emphasis added):
Let’s go back to what Carl said:
Sure sounds like ‘Marcus Hutter...have written about the future of AI and risk of human extinction’.
Which in that case demonstrates awareness among the AI researchers of the risk, while at the same time not demonstrating that Hutter finds it particularly likely that this would happen (‘might’) or agrees with any specific alarmist rhetoric. I can’t know if that’s what Carl actually refers to. I do assure you that about every AI researcher has seen the Terminator.
I gave the Hutter quote I was thinking of upthread.
My aim was basically to distinguish between buying Eliezer’s claims and taking intelligence explosion and AI risk seriously, and to reject the idea that the ideas in question came out of nowhere. One can think AI risk is worth investigating without thinking much of Eliezer’s views or SI.
I agree that the cited authors would assign much lower odds of catastrophe given human-level AI than Eliezer. The same statement would be true of myself, or of most people at SI and FHI: Eliezer is at the far right tail on those views. Likewise for the probability that a small team assembled in the near future could build safe AGI first, but otherwise catastrophe would have ensued.
Well, I guess that’s fair enough. In the quote on the top, though, I am specifically criticizing the extreme view. At the end of the day, the entire raison d’etre for SI’s existence is the claim that without paying you the risk would be higher. The claim that you are somehow fairy unique. And there are many risks—for example, risk of lethal flu-like pandemic—which are much more clearly understood and where specific efforts have much more clearly predictable outcome of reducing the risk. Favouring a group of AI theorists but not other does not have clearly predictable outcome of reducing the risk.
(I am inclined to believe that the pandemic is under funded as it would primarily decimate the poorer countries, ending existence of entire cultures, whereas the ‘existential risk’ is a fancy phrase for a risk to the privileged)
It need not demonstrate any such thing to fit Carl’s statement perfectly and give the lie to your claim that he was misrepresenting Hutter.
Sure, hence the Hutter citation of “(Cameron 1984)”. Oh wait.
The argument is for the insights coming out of EY , and the privileging that EY is making for those hypotheses originated by others, aka cherrypicking what to advertise. EY is a good writer.
edit: concrete thought example: There is a drug A that undergoes many tests, with some of them evaluating it as better than placebo, some as equal to placebo, and some as worse to placebo. Worst of all, each trial is conducted on 1 person’s opinion. Comes in the charismatic pharmaceutical marketer, or charismatic anti-vaccination campaign leader, and starts bringing to attention the negative or positive trials. That is not good. Even if there’s both of those people.