This is a definitely an assumption that should be challenged more. However, I don’t think that FOOM is remotely required for a lot of AI X-risk (or at least unprecedented catastrophic human death toll risk) scenarios. Something doesn’t need to recursively self-improve to be a threat if it’s given powerful enough ways to act on the world (and all signs point to us being exactly dumb enough to do that). All that’s required is that we aren’t able to coordinate well enough as a species to actually stop it. Either we don’t detect the threat before it’s too late or we aren’t able to get someone to actually hit the “off” button (literally or figuratively) in time if the threat is detected. And if it only kills 90% of humans because of some error and doesn’t tile its light cone in paperclips, that’s still really, really bad from a human perspective.
All that’s required is that we aren’t able to coordinate well enough as a species to actually stop it.
Indeed I would be much more optimistic if we were better at dealing with much simpler challenges, like put a price on pollution and welcome refugees with humanity.
This is a definitely an assumption that should be challenged more. However, I don’t think that FOOM is remotely required for a lot of AI X-risk (or at least unprecedented catastrophic human death toll risk) scenarios. Something doesn’t need to recursively self-improve to be a threat if it’s given powerful enough ways to act on the world (and all signs point to us being exactly dumb enough to do that). All that’s required is that we aren’t able to coordinate well enough as a species to actually stop it. Either we don’t detect the threat before it’s too late or we aren’t able to get someone to actually hit the “off” button (literally or figuratively) in time if the threat is detected. And if it only kills 90% of humans because of some error and doesn’t tile its light cone in paperclips, that’s still really, really bad from a human perspective.
Indeed I would be much more optimistic if we were better at dealing with much simpler challenges, like put a price on pollution and welcome refugees with humanity.