One possible framing is hubristic overconfidence vs. humble caution.
We aren’t saying AGI will overthrow humanity soon if we’re not careful; we’re saying it could. Everyone saying that’s ridiculous is essentially saying “hold my beer!” while attempting a stunt nobody has ever pulled off before. They could be right that it will be easy enough, they’re smart enough, and they will know when the danger approaches. But they’re gambling the future of all humanity on that confidence.
Experts have widely varying opinions on the dangers of AGI, so the humble belief is that we don’t know what’s possible, so should behave in accordance with accepting a very broad distribution of timelines, alignment difficulty, and levels of coordination.
That framing won’t make the most concerned among us happy. It will result in people who aren’t long-termists wanting to approach AGI fast enough to save themselves or their children from a painful death from natural causes. But it might be an acceptable compromise, while we gather and analyze more information.
One possible framing is hubristic overconfidence vs. humble caution.
We aren’t saying AGI will overthrow humanity soon if we’re not careful; we’re saying it could. Everyone saying that’s ridiculous is essentially saying “hold my beer!” while attempting a stunt nobody has ever pulled off before. They could be right that it will be easy enough, they’re smart enough, and they will know when the danger approaches. But they’re gambling the future of all humanity on that confidence.
Experts have widely varying opinions on the dangers of AGI, so the humble belief is that we don’t know what’s possible, so should behave in accordance with accepting a very broad distribution of timelines, alignment difficulty, and levels of coordination.
That framing won’t make the most concerned among us happy. It will result in people who aren’t long-termists wanting to approach AGI fast enough to save themselves or their children from a painful death from natural causes. But it might be an acceptable compromise, while we gather and analyze more information.