Assuming slower and more gradual timelines, isn’t it likely that we run into some smaller, more manageable AI catastrophes before “everybody falls over dead” due to the first ASI going rogue? Maybe we’ll be at a state of sub-human level AGIs for a while, and during that time some of the AIs clearly demonstrate misaligned behavior leading to casualties (and general insights into what is going wrong), in turn leading to a shift in public perception. Of course it might still be unlikely that the whole globe at that point stops improving AIs and/or solves alignment in time, but it would at least push awareness and incentives somewhat into the right direction.
Assuming slower and more gradual timelines, isn’t it likely that we run into some smaller, more manageable AI catastrophes before “everybody falls over dead” due to the first ASI going rogue? Maybe we’ll be at a state of sub-human level AGIs for a while, and during that time some of the AIs clearly demonstrate misaligned behavior leading to casualties (and general insights into what is going wrong), in turn leading to a shift in public perception. Of course it might still be unlikely that the whole globe at that point stops improving AIs and/or solves alignment in time, but it would at least push awareness and incentives somewhat into the right direction.
This does seem very possible if you assume a slower takeoff.
This is the most likely scenario, with AGI getting heavily regulated, similarly to nuclear. It doesn’t get much publicity because it’s “boring”.