I’m wondering what Nick Bostrom’s p(doom) currently is, given the subject of this book. He said 9 years ago in his lecture on his book Superintelligence “less than 50% risk of doom”. In this interview 4 months ago he said that it’s good there has been more focus on risks in recent times, but there’s still slightly less focus on the risks than what is optimal, but he wants to focus on the upsides because he fears we might “overshoot” and not build AGI at all which would be tragic in his opinion. So it seems he thinks the risk is less than it used to be because of this public awareness of the risks.
In Bostrom’s recent interview with Liv Boeree, he said (I’m paraphrasing; you’re probably better off listening to what he actually said)
p(doom)-related
it’s actually gone up for him, not down (contra your guess, unless I misinterpreted you), at least when broadening the scope beyond AI (cf. vulnerable world hypothesis, 34:50 in video)
re: AI, his prob. dist. has ‘narrowed towards the shorter end of the timeline—not a huge surprise, but a bit faster I think’ (30:24 in video)
also re: AI, ‘slow and medium-speed takeoffs have gained credibility compared to fast takeoffs’
he wouldn’t overstate any of this
contrary to people’s impression of him, he’s always been writing about ‘both sides’ (doom and utopia)
in the past it just seemed more pressing to him to call attention to ‘various things that could go wrong so we could avoid these pitfalls and then we’d have plenty of time to think about what to do with this big future’
this reminded me of this illustration from his old paper introducing the idea of x-risk prevention as global priority:
It’s very surprising to me that he would think there’s a real chance of all humans collectively deciding to not build AGI, and successfully enforcing the ban indefinitely.
I’m wondering what Nick Bostrom’s p(doom) currently is, given the subject of this book. He said 9 years ago in his lecture on his book Superintelligence “less than 50% risk of doom”. In this interview 4 months ago he said that it’s good there has been more focus on risks in recent times, but there’s still slightly less focus on the risks than what is optimal, but he wants to focus on the upsides because he fears we might “overshoot” and not build AGI at all which would be tragic in his opinion. So it seems he thinks the risk is less than it used to be because of this public awareness of the risks.
In Bostrom’s recent interview with Liv Boeree, he said (I’m paraphrasing; you’re probably better off listening to what he actually said)
p(doom)-related
it’s actually gone up for him, not down (contra your guess, unless I misinterpreted you), at least when broadening the scope beyond AI (cf. vulnerable world hypothesis, 34:50 in video)
re: AI, his prob. dist. has ‘narrowed towards the shorter end of the timeline—not a huge surprise, but a bit faster I think’ (30:24 in video)
also re: AI, ‘slow and medium-speed takeoffs have gained credibility compared to fast takeoffs’
he wouldn’t overstate any of this
contrary to people’s impression of him, he’s always been writing about ‘both sides’ (doom and utopia)
e.g. his Letter from Utopia first published in 2005,
in the past it just seemed more pressing to him to call attention to ‘various things that could go wrong so we could avoid these pitfalls and then we’d have plenty of time to think about what to do with this big future’
this reminded me of this illustration from his old paper introducing the idea of x-risk prevention as global priority:
It seems that in 2014 he believed that p(doom) was less than 20%
It’s very surprising to me that he would think there’s a real chance of all humans collectively deciding to not build AGI, and successfully enforcing the ban indefinitely.