One could offer many crude back-of-envelope probability calculations. Here’s one: let’s say there’s
a 10% chance AGI is easy enough for the world to do in the next few decades
a 1% chance that if the world can do it, a team of supergeniuses can do the Friendly kind first
an independent 10% chance Eliezer succeeds at putting together such a team of supergeniuses
That seems conservative to me and implies at least a 1 in 10^4 chance. Obviously there’s lots of room for quibbling here, but it’s hard for me to see how such quibbling could account for five orders of magnitude. And even if post-quibbling you think you have a better model that does imply 1 in 10^9, you only need to put little probability mass on my model or models like it for them to dominate the calculation. (E.g., a 9 in 10 chance of a 1 in 10^9 chance plus a 1 in 10 chance of a 1 in 10^4 chance is close to a 1 in 10^5 chance.)
I don’t find these remarks compelling. I feel similar remarks could be used to justify nearly anything. Of course, I owe you an explanation. One will follow later on.
One could offer many crude back-of-envelope probability calculations. Here’s one: let’s say there’s
a 10% chance AGI is easy enough for the world to do in the next few decades
a 1% chance that if the world can do it, a team of supergeniuses can do the Friendly kind first
an independent 10% chance Eliezer succeeds at putting together such a team of supergeniuses
That seems conservative to me and implies at least a 1 in 10^4 chance. Obviously there’s lots of room for quibbling here, but it’s hard for me to see how such quibbling could account for five orders of magnitude. And even if post-quibbling you think you have a better model that does imply 1 in 10^9, you only need to put little probability mass on my model or models like it for them to dominate the calculation. (E.g., a 9 in 10 chance of a 1 in 10^9 chance plus a 1 in 10 chance of a 1 in 10^4 chance is close to a 1 in 10^5 chance.)
I don’t find these remarks compelling. I feel similar remarks could be used to justify nearly anything. Of course, I owe you an explanation. One will follow later on.