That’s a fair point—my model does assume AGI will come into existence in non-negative worlds. Though I struggle to actually imagine a non-negative world where humanity is alive a thousand years from now and AGI hasn’t been developed. Even if all alignment researchers believed it was the right thing to pursue, which doesn’t seem likely.
That’s not a middle ground between a good world and a neutral world, though, that’s just another way to get a good world. If we assume a good world is exponentially unlikely, a 10 year delay might mean the odds of a good world rise from 10^-10 to 10^-8 (as opposed to pursuing Clippy bringing the odds of a bad world down from 10^-4 to 10^-6 ).
If you disagree with Yudkowsky about his pessimism about the probability of good worlds, then my post doesn’t really apply. My post is about how to handle him being correct about the odds.
That’s a fair point—my model does assume AGI will come into existence in non-negative worlds. Though I struggle to actually imagine a non-negative world where humanity is alive a thousand years from now and AGI hasn’t been developed. Even if all alignment researchers believed it was the right thing to pursue, which doesn’t seem likely.
Even a 5~10 year delay in AGI deployment might give enough time to solve the alignment problem.
That’s not a middle ground between a good world and a neutral world, though, that’s just another way to get a good world. If we assume a good world is exponentially unlikely, a 10 year delay might mean the odds of a good world rise from 10^-10 to 10^-8 (as opposed to pursuing Clippy bringing the odds of a bad world down from 10^-4 to 10^-6 ).
If you disagree with Yudkowsky about his pessimism about the probability of good worlds, then my post doesn’t really apply. My post is about how to handle him being correct about the odds.