I have a non-zero probability for ASI in my lifetime, and I treat it in the same fashion I would risks like “Maybe we all get killed by a magnetar crust quake a thousand light-years away” or “Maybe I die in a car accident.” I plan as if worst won’t come to worst, and I try to use the time I have better.
Conditional on actually building ASI, my P(doom) + P(pets) is greater than 95%, where P(pets) covers scenarios like “a vastly super-human AI keeps us around out of nostalgia or a sense of ethics, but we’re not in control in any meaningful sense.” Scenarios like the Culture and CelestAI would fall under P(pets). And “successful” alignment work just shifts some probability mass from P(doom) to P(pets), or improves the P(pets) outcomes to slightly better versions of the same idea.
And even though I don’t really believe in scenarios like “An ASI obeys a specific human or group of humans,” or “An ASI obeys the democratic will of humanity”, I suspect that most of those hypothetical outcomes would be deeply dystopian at best, and very possibly worse on average than P(pets).
Which leaves two major questions, P(ASI|AGI), and the timelines for each step. I would be surprised by very short timelines (ASI in 2027), but I would also be at least mildly surprised not to see ASI within 100 years.
So I wish we would stop banging together large hunks plutonium to see what happens. But if we get even close to AGI, I expect governance structures to break down completely, and the final human decisions to most likely be made by sociopaths, fools and people who imagine that they are slaves to competitive forces.
So overall, I plan for the possibility of ASI the same way I would plan for personal death or a planetary-scale natural disaster with 100% mortality but unknown probability of occurring. I don’t plan for P(pets), because it’s not like my decisions now would have much influence on what happens in such a scenario.
I have a non-zero probability for ASI in my lifetime, and I treat it in the same fashion I would risks like “Maybe we all get killed by a magnetar crust quake a thousand light-years away” or “Maybe I die in a car accident.” I plan as if worst won’t come to worst, and I try to use the time I have better.
Conditional on actually building ASI, my P(doom) + P(pets) is greater than 95%, where P(pets) covers scenarios like “a vastly super-human AI keeps us around out of nostalgia or a sense of ethics, but we’re not in control in any meaningful sense.” Scenarios like the Culture and CelestAI would fall under P(pets). And “successful” alignment work just shifts some probability mass from P(doom) to P(pets), or improves the P(pets) outcomes to slightly better versions of the same idea.
And even though I don’t really believe in scenarios like “An ASI obeys a specific human or group of humans,” or “An ASI obeys the democratic will of humanity”, I suspect that most of those hypothetical outcomes would be deeply dystopian at best, and very possibly worse on average than P(pets).
Which leaves two major questions, P(ASI|AGI), and the timelines for each step. I would be surprised by very short timelines (ASI in 2027), but I would also be at least mildly surprised not to see ASI within 100 years.
So I wish we would stop banging together large hunks plutonium to see what happens. But if we get even close to AGI, I expect governance structures to break down completely, and the final human decisions to most likely be made by sociopaths, fools and people who imagine that they are slaves to competitive forces.
So overall, I plan for the possibility of ASI the same way I would plan for personal death or a planetary-scale natural disaster with 100% mortality but unknown probability of occurring. I don’t plan for P(pets), because it’s not like my decisions now would have much influence on what happens in such a scenario.