Eliezer, thanks for sharing these ideas so that more people can be on the lookout for failures. Personally, I think something like 15% of AGI dev teams (weighted by success probability) would destroy the world more-or-less immediately, and I think it’s not crazy to think the fraction is more like 90% or higher (which I judge to be your view).
FWIW, I do not agree with the following stance, because I think it exposes the world to more x-risk:
Specifically, I think a considerable fraction of the remaining AI x-risk facing humanity stems from people pulling desperate (unsafe) moves with AGI to head off other AGI projects. So, in that regard, I think that particular comment of yours is probably increasing x-risk a bit. If I were a 90%-er like you, it’s possible I’d endorse it, but even then it might make things worse by encouraging more desperate unilateral actions.
That said, overall I think this post is a big help, because it helps to put responsibility in the hands of more people to not do the crazy/stupid/reckless things you’re describing here… and while I might disagree on the fraction/probability, I agree that some groups would destroy humanity more or less immediately if they developed AGI. And, while I might disagree on some of the details of how human extinction eventually plays out, I do think human extinction remains the default outcome of humanity’s path toward replacing itself with automation, probably within our lifetimes unfortunately.
a considerable fraction of the remaining AI x-risk facing humanity stems from people pulling desperate (unsafe) moves with AGI to head off other AGI projects
In your post “Pivotal Act” Intentions, you wrote that you disagree with contributing to race dynamics by planning to invasively shut down AGI projects because AGI projects would, in reaction, try to maintain
the ability to implement their own pet theories on how safety/alignment should work, leading to more desperation, more risk-taking, and less safety overall.
Could you give some kind of very rough estimates here? How much more risk-taking do you expect in a world given how much / how many prominent “AI safety”-affiliated people declaring invasive pivotal act intentions? How much risk-taking do you expect in the alternative, where there are other pressures (economic, military, social, whatever), but not pressure from pivotal act threats? How much safety (probability of AGI not killing everyone) do you think this buys? You write:
15% of AGI dev teams (weighted by success probability) would destroy the world more-or-less immediately
Eliezer, thanks for sharing these ideas so that more people can be on the lookout for failures. Personally, I think something like 15% of AGI dev teams (weighted by success probability) would destroy the world more-or-less immediately, and I think it’s not crazy to think the fraction is more like 90% or higher (which I judge to be your view).
FWIW, I do not agree with the following stance, because I think it exposes the world to more x-risk:
Specifically, I think a considerable fraction of the remaining AI x-risk facing humanity stems from people pulling desperate (unsafe) moves with AGI to head off other AGI projects. So, in that regard, I think that particular comment of yours is probably increasing x-risk a bit. If I were a 90%-er like you, it’s possible I’d endorse it, but even then it might make things worse by encouraging more desperate unilateral actions.
That said, overall I think this post is a big help, because it helps to put responsibility in the hands of more people to not do the crazy/stupid/reckless things you’re describing here… and while I might disagree on the fraction/probability, I agree that some groups would destroy humanity more or less immediately if they developed AGI. And, while I might disagree on some of the details of how human extinction eventually plays out, I do think human extinction remains the default outcome of humanity’s path toward replacing itself with automation, probably within our lifetimes unfortunately.
In your post “Pivotal Act” Intentions, you wrote that you disagree with contributing to race dynamics by planning to invasively shut down AGI projects because AGI projects would, in reaction, try to maintain
Could you give some kind of very rough estimates here? How much more risk-taking do you expect in a world given how much / how many prominent “AI safety”-affiliated people declaring invasive pivotal act intentions? How much risk-taking do you expect in the alternative, where there are other pressures (economic, military, social, whatever), but not pressure from pivotal act threats? How much safety (probability of AGI not killing everyone) do you think this buys? You write:
What about non-immediately, in each alternative?