The “Respondents’ comments” section of the existential risk survey I ran last year gives some examples of people’s reasoning for different risk levels. My own p(doom) is more like 99%, so I don’t want to speak on behalf of people who are less worried. Relevant factors, thought, include:
Specific reasons to think things may go well. (I gave some of my own here.)
Disagreement with various points in AGI Ruin. E.g., I think a lot of EAs believe some combination of:
The alignment problem plausibly isn’t very hard. (E.g., maybe we can just give the AGI/TAI a bunch of training data indicating that obedient, deferential, low-impact, and otherwise corrigible behavior is good, and then this will generalize fine in practice without our needing to do anything special.)
The field of alignment research has grown fast, and has had lots of promising ideas already.
AGI/TAI is probably decades away, and progress toward it will probably be gradual. This gives plenty of time for more researchers to notice “we’re getting close” and contribute to alignment research, and for the field in general to get a lot more serious about AI risk.
Another consequence of ‘AI progress is gradual’: Insofar as AI is very dangerous or hard to align, we can expect that there will be disasters like “AI causes a million deaths” well before there are disasters like “AI kills all humans”. The response to disasters like “a million deaths” (both on the part of researchers and on the part of policymakers, etc.) would probably be reasonable and helpful, especially with EAs around to direct the response in good directions. So we can expect the response to get better and better as we get closer to transformative AI.
General skepticism about our ability to predict the future with any confidence. Even if you aren’t updating much on ‘most past doom predictions were wrong’, you should have less extreme probabilities insofar as you think it’s harder to predict stuff in general.
The “Respondents’ comments” section of the existential risk survey I ran last year gives some examples of people’s reasoning for different risk levels. My own p(doom) is more like 99%, so I don’t want to speak on behalf of people who are less worried. Relevant factors, thought, include:
Specific reasons to think things may go well. (I gave some of my own here.)
Disagreement with various points in AGI Ruin. E.g., I think a lot of EAs believe some combination of:
The alignment problem plausibly isn’t very hard. (E.g., maybe we can just give the AGI/TAI a bunch of training data indicating that obedient, deferential, low-impact, and otherwise corrigible behavior is good, and then this will generalize fine in practice without our needing to do anything special.)
The field of alignment research has grown fast, and has had lots of promising ideas already.
AGI/TAI is probably decades away, and progress toward it will probably be gradual. This gives plenty of time for more researchers to notice “we’re getting close” and contribute to alignment research, and for the field in general to get a lot more serious about AI risk.
Another consequence of ‘AI progress is gradual’: Insofar as AI is very dangerous or hard to align, we can expect that there will be disasters like “AI causes a million deaths” well before there are disasters like “AI kills all humans”. The response to disasters like “a million deaths” (both on the part of researchers and on the part of policymakers, etc.) would probably be reasonable and helpful, especially with EAs around to direct the response in good directions. So we can expect the response to get better and better as we get closer to transformative AI.
General skepticism about our ability to predict the future with any confidence. Even if you aren’t updating much on ‘most past doom predictions were wrong’, you should have less extreme probabilities insofar as you think it’s harder to predict stuff in general.