A utilitarian, a deep ecologist, and a Christian might agree on policy in the present world, but given arbitrary power their preferred futures might be a radical loss to the others. <...> People who broadly agree on good outcomes within the current world may, given much more power, choose outcomes that others would consider catastrophic
I think that many people do not intend for their preferred policy to be implemented everywhere; so, at least they could be satisfied with a small region of universe. Though, AI-as-tool is quite likely to be created under control of those who want to have every power source and (an assumption) thus also want to steer most of the world; it’s unclear if AI-as-agent would have strong preferences about the parts of world it doesn’t see.
“Everyone who only cares about their slices of the world coordinates against those who want to seize control of the entire world” seems like it might be one of those stable equilibria.
Which is why, since the beginning of the nuclear age, the running theme of international relations is “a single nation embarked on multiple highly destructive wars of conquest, and continued along those lines until no nations that could threaten it remained”.
I think that many people do not intend for their preferred policy to be implemented everywhere; so, at least they could be satisfied with a small region of universe. Though, AI-as-tool is quite likely to be created under control of those who want to have every power source and (an assumption) thus also want to steer most of the world; it’s unclear if AI-as-agent would have strong preferences about the parts of world it doesn’t see.
“Everyone who only cares about their slices of the world coordinates against those who want to seize control of the entire world” seems like it might be one of those stable equilibria.
Alas, the side that wants to win always beats the side that just wants to be left alone.
Which is why, since the beginning of the nuclear age, the running theme of international relations is “a single nation embarked on multiple highly destructive wars of conquest, and continued along those lines until no nations that could threaten it remained”.