From my perspective, there’s an important additional level: Economic pressure.
Given that it takes time for messages to travel through the world, there will likely be multiple AGIs in scenarios where there is no pivotal act that limits the world to a single AGI.
Those AGIs that highly prioritize getting more power and resources are likely going to get more power than AGIs that don’t. Competition between AGIs can be more fierce than competition between humans as a human can’t simply duplicate themselves by eating twice as much and raising new children is a time-consuming process but an AGI can use resources that are used by other AGIs to spin up more copies of itself.
In a world where most of the power is held by AGIs valuing human values is an impediment to prioritizing resource acquisition. As AGIs evolve and likely go through core ontological shifts there’s selection pressure toward deemphasizing human flourishing.
Given that it takes time for messages to travel through the world, there will likely be multiple AGIs
If AGIs can’t build subagents without having those rebel against their master, I don’t think they’ll install subagents across the world to save a quarter second on ping.
I do think that there’s a reasonable possibility that there will be multiple not-fully-human-controlled AGIs competing against each other for various forms of power. I don’t think the specific scenario you outline seems like a particularly plausible way to get there. Also, I think humanity has a lot more leverage before that situation comes to pass, so I believe we will get more ‘expected value per unit of effort’ if we focus our safety planning on preventing ‘multiple poorly controlled AGIs competing’ rather than dealing with that.
From my perspective, there’s an important additional level: Economic pressure.
Given that it takes time for messages to travel through the world, there will likely be multiple AGIs in scenarios where there is no pivotal act that limits the world to a single AGI.
Those AGIs that highly prioritize getting more power and resources are likely going to get more power than AGIs that don’t. Competition between AGIs can be more fierce than competition between humans as a human can’t simply duplicate themselves by eating twice as much and raising new children is a time-consuming process but an AGI can use resources that are used by other AGIs to spin up more copies of itself.
In a world where most of the power is held by AGIs valuing human values is an impediment to prioritizing resource acquisition. As AGIs evolve and likely go through core ontological shifts there’s selection pressure toward deemphasizing human flourishing.
If AGIs can’t build subagents without having those rebel against their master, I don’t think they’ll install subagents across the world to save a quarter second on ping.
I do think that there’s a reasonable possibility that there will be multiple not-fully-human-controlled AGIs competing against each other for various forms of power. I don’t think the specific scenario you outline seems like a particularly plausible way to get there. Also, I think humanity has a lot more leverage before that situation comes to pass, so I believe we will get more ‘expected value per unit of effort’ if we focus our safety planning on preventing ‘multiple poorly controlled AGIs competing’ rather than dealing with that.