Nobody I have ever met outside of the EA sphere seriously believes that superintelligent computer systems could take over the world within decades.
A lot of prominent scientists, technologists and intellectuals outside of EA have warned about advanced artificial intelligence too. Stephen Hawking, Elon Musk, Bill Gates, Sam Harris, everyone on this open letter back in 2015 etc.
I agree that the number of people really concerned about this is strikingly small given the emphasis longtermist EAs put on it. But I think these many counter-examples warn us that it’s not just EAs and the AGI labs being overconfident or out of left field.
I know you said you don’t have time to fully debate this. This seemed to be one of the cruxes of your first bullet point though. So if your skepticism about short timelines is driven in a big way by thinking that no credible person outside EA or companies invested in AI think this is plausible, then I am curious what you make of this.
Hey Evan, thanks for the response. You’re right that there are circles where short AI timelines are common. My comment was specifically about people I personally know, which is absolutely not the best reference class. Let me point out a few groups with various clusters of timelines.
Artificial intelligence researchers are a group of people who believe in short to medium AI timelines. Katja Grace’s 2015 survey of NIPS and ICML researchers provided an aggregate forecast giving a 50% chance of HLMI occurring by 2060 and a 10% chance of it occurring by 2024. (Today, seven years after the survey was conducted, you might want to update against the researchers that predicted HLMI by 2024.) Other surveys of ML researchers have shown similarly short timelines. This seems as good of an authority as any on the topic, and would be one of the better reasons to have relatively short timelines.
What I’ll call the EA AI Safety establishment has similar timelines to the above. This would include decision makers at OpenPhil, OpenAI, FHI, FLI, CHAI, ARC, Redwood, Anthropic, Ought, and other researchers and practitioners of AI safety work. As best I can tell, Holden Karnofsky’s timelines are reasonably similar to the others in this reference group, including Paul Christiano and Rohin Shah (would love to add more examples if anybody can point to them), although I’m sure there are plenty of individual outliers. I have a bit longer timelines than most of these people for a few object level reasons, but their timelines seem reasonable.
Much shorter timelines than the two groups above come from Eliezer Yudkowsky, MIRI, many people on LessWrong, and others. You can read this summary of Yudkowsky’s conversation with Paul Christiano, where he does not quantify his timelines but consistently argues for faster takeoff speeds than Christiano believes are likely. See also this aggregation of the five most upvoted timelines from LW users, with a median of 25 years until AGI. That is 15 years sooner than Holden Karnofsky and 15 years sooner than Katja Grace’s survey of ML researchers. This is the group of scenarios that I would most strongly disagree with, appealing to both the “expert” consensus and my object level arguments above.
The open letter from FLI does not mention any specific AI timelines at all. These individuals all agree that the dangers from AI are significant and that AI safety research is important, but I don’t believe most of them have particularly short timelines. You can read about Bill Gates’s timelines here, he benchmarks his timelines as “at least 5 times as long as what Ray Kurzweil says”. I’m sure other signatories of the letter have talked about their timelines, I’d love to add these quotes but haven’t found any others.
Overall, I’d still point to Holden Karnofsky’s estimates as the most reasonable “consensus” on the topic. The object-level reasons I’ve outlined above are part of the reason why I have longer timelines than Holden, but even without those, I don’t think it’s reasonable to “pull the short timelines fire alarm”.
Katja Grace’s 2015 survey of NIPS and ICML researchers provided an aggregate forecast giving a 50% chance of HLMI occurring by 2060 and a 10% chance of it occurring by 2024.
2015 feels decades ago though. That’s before GPT-1!
(Today, seven years after the survey was conducted, you might want to update against the researchers that predicted HLMI by 2024.)
I would expect a survey done today to have more researchers predicting 2024. Certainly I’d expect a median before 2060! My layman impression is that things have turned out to be easier to do for big language models, not harder.
This was heavily upvoted at the time of posting, including by me. It turns out to be mostly wrong. AI Impacts just released a survey of 4271 NeurIPS and ICML researchers conducted in 2021 and found that the median year for expected HLMI is 2059, down only two years from 2061 since 2016. Looks like the last five years of evidence hasn’t swayed the field much. My inside view says they’re wrong, but the opinions of the field and our inability to anticipate them are both important.
Appreciate the super thorough response, Aidan. You’re right it turns out that some of the people I mentioned like Gates who are concerned about AI aren’t on record with particularly short timelines.
I agree with FeepingCreature’s comment that some of these surveys and sources are starting to feel quite dated. Apparently GovAI is currently working on replicating the 2015 Grace et al. survey which I’m very much looking forward to. The Bostrom survey you linked to is even older than that—from 2012~13. At least the Gruetzemacher survey is from 2018.
(tangent) One thing I see in some of the surveys as well as in discussion that bothers me is an emphasis on automation of 100% of tasks or 99% of tasks. I think Holden made an important point in his discussion of a weak point in his Most Important Century series, that transformative AI need not depend on everything being automated. In fact, just having full-automation of a small number of specific activities could be all that’s needed:
...it’s also worth noting that the extreme levels of automation need not apply to the whole economy: extreme automation for a relatively small set of activities could be sufficient to reach the conclusions in the series.
For example, it might be sufficient for AI systems to develop increasingly efficient (a) computers; (b) solar panels (for energy); (c) mining and manufacturing robots; (d) space probes (to build more computers in space, where energy and metal are abundant). That could be sufficient (via feedback loop) for explosive growth in available energy, materials and computing power, and there are many ways that such growth could be transformative.
For example and in particular, it could lead to:
Misaligned AI with access to dangerous amounts of materials and energy.
Pulling from those comments, you said:
A lot of prominent scientists, technologists and intellectuals outside of EA have warned about advanced artificial intelligence too. Stephen Hawking, Elon Musk, Bill Gates, Sam Harris, everyone on this open letter back in 2015 etc.
I agree that the number of people really concerned about this is strikingly small given the emphasis longtermist EAs put on it. But I think these many counter-examples warn us that it’s not just EAs and the AGI labs being overconfident or out of left field.
I know you said you don’t have time to fully debate this. This seemed to be one of the cruxes of your first bullet point though. So if your skepticism about short timelines is driven in a big way by thinking that no credible person outside EA or companies invested in AI think this is plausible, then I am curious what you make of this.
Hey Evan, thanks for the response. You’re right that there are circles where short AI timelines are common. My comment was specifically about people I personally know, which is absolutely not the best reference class. Let me point out a few groups with various clusters of timelines.
Artificial intelligence researchers are a group of people who believe in short to medium AI timelines. Katja Grace’s 2015 survey of NIPS and ICML researchers provided an aggregate forecast giving a 50% chance of HLMI occurring by 2060 and a 10% chance of it occurring by 2024. (Today, seven years after the survey was conducted, you might want to update against the researchers that predicted HLMI by 2024.) Other surveys of ML researchers have shown similarly short timelines. This seems as good of an authority as any on the topic, and would be one of the better reasons to have relatively short timelines.
What I’ll call the EA AI Safety establishment has similar timelines to the above. This would include decision makers at OpenPhil, OpenAI, FHI, FLI, CHAI, ARC, Redwood, Anthropic, Ought, and other researchers and practitioners of AI safety work. As best I can tell, Holden Karnofsky’s timelines are reasonably similar to the others in this reference group, including Paul Christiano and Rohin Shah (would love to add more examples if anybody can point to them), although I’m sure there are plenty of individual outliers. I have a bit longer timelines than most of these people for a few object level reasons, but their timelines seem reasonable.
Much shorter timelines than the two groups above come from Eliezer Yudkowsky, MIRI, many people on LessWrong, and others. You can read this summary of Yudkowsky’s conversation with Paul Christiano, where he does not quantify his timelines but consistently argues for faster takeoff speeds than Christiano believes are likely. See also this aggregation of the five most upvoted timelines from LW users, with a median of 25 years until AGI. That is 15 years sooner than Holden Karnofsky and 15 years sooner than Katja Grace’s survey of ML researchers. This is the group of scenarios that I would most strongly disagree with, appealing to both the “expert” consensus and my object level arguments above.
The open letter from FLI does not mention any specific AI timelines at all. These individuals all agree that the dangers from AI are significant and that AI safety research is important, but I don’t believe most of them have particularly short timelines. You can read about Bill Gates’s timelines here, he benchmarks his timelines as “at least 5 times as long as what Ray Kurzweil says”. I’m sure other signatories of the letter have talked about their timelines, I’d love to add these quotes but haven’t found any others.
Overall, I’d still point to Holden Karnofsky’s estimates as the most reasonable “consensus” on the topic. The object-level reasons I’ve outlined above are part of the reason why I have longer timelines than Holden, but even without those, I don’t think it’s reasonable to “pull the short timelines fire alarm”.
2015 feels decades ago though. That’s before GPT-1!
I would expect a survey done today to have more researchers predicting 2024. Certainly I’d expect a median before 2060! My layman impression is that things have turned out to be easier to do for big language models, not harder.
The surveys urgently need to be updated.
This was heavily upvoted at the time of posting, including by me. It turns out to be mostly wrong. AI Impacts just released a survey of 4271 NeurIPS and ICML researchers conducted in 2021 and found that the median year for expected HLMI is 2059, down only two years from 2061 since 2016. Looks like the last five years of evidence hasn’t swayed the field much. My inside view says they’re wrong, but the opinions of the field and our inability to anticipate them are both important.
https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/
Appreciate the super thorough response, Aidan. You’re right it turns out that some of the people I mentioned like Gates who are concerned about AI aren’t on record with particularly short timelines.
I agree with FeepingCreature’s comment that some of these surveys and sources are starting to feel quite dated. Apparently GovAI is currently working on replicating the 2015 Grace et al. survey which I’m very much looking forward to. The Bostrom survey you linked to is even older than that—from 2012~13. At least the Gruetzemacher survey is from 2018.
(tangent) One thing I see in some of the surveys as well as in discussion that bothers me is an emphasis on automation of 100% of tasks or 99% of tasks. I think Holden made an important point in his discussion of a weak point in his Most Important Century series, that transformative AI need not depend on everything being automated. In fact, just having full-automation of a small number of specific activities could be all that’s needed: