My main default prediction here is that we will avoid either the absolute best case or the absolute worst case scenario, because I predict intent alignment works well enough to avoid extinction of humanity type scenarios, but I also don’t believe we will see radical movements toward equality (indeed the politics of our era is moving towards greater acceptance of inequality), so capitalism more or less survives the transition to AGI.
I do think dynamism will still exist, but it will be very limited to the upper classes/very rich of society, and most people will not be a part of it, and I’m including uploaded humans here in this calculation.
To address this:
Rationalist thought on post-AGI futures is too solutionist. The strawman version: solve morality, solve AI, figure out the optimal structure to tile the universe with, do that, done. (The actual leading figures have far less strawman views; see e.g. Paul Christiano at 23:30 here—but the on-the-ground culture does lean in the strawman direction.)
To be somewhat more fair, the worry here is that in a regime where you don’t need society anymore because AIs can do all the work for your society, value conflicts become a bigger deal than today, because there is less reason to tolerate other people’s values if you can just found your own society based on your own values, and if you believe in the vulnerable world hypothesis, as a lot of rationalists do, then conflict has existential stakes, and even if not, can be quite bad, so one group controlling the future is better than inevitable conflict.
At a foundational level, whether or not our current tolerance for differing values is stable ultimately comes down to we can compensate for the effect of AGI allowing people to make their own society.
(indeed the politics of our era is moving towards greater acceptance of inequality)
How certain are you of this, and how much do you think it comes down more to something like “to what extent can disempowered groups unionise against the elite?”.
To be clear, by default I think AI will make unionising against the more powerful harder, but it might depend on the governance structure. Maybe if we are really careful, we can get something closer to “Direct Democracy”, where individual preferences actually matter more!
I am focused here on short-term politics in the US, which ordinarily would matter less, if it wasn’t likely that world-changing AI would be built in the US, but given that it might, it becomes way more important than normal.
To be somewhat more fair, the worry here is that in a regime where you don’t need society anymore because AIs can do all the work for your society, value conflicts become a bigger deal than today, because there is less reason to tolerate other people’s values if you can just found your own society based on your own values, and if you believe in the vulnerable world hypothesis, as a lot of rationalists do, then conflict has existential stakes, and even if not, can be quite bad, so one group controlling the future is better than inevitable conflict.
So to summarise: if we have a multipolar world, and the vulnerable world hypothesis if true, then conflict can be existentially bad and this is a reason to avoid a multipolar world. Didn’t consider this, interesting point!
At a foundational level, whether or not our current tolerance for differing values is stable ultimately comes down to we can compensate for the effect of AGI allowing people to make their own society.
Considerations:
offense/defense balance (if offense wins very hard, it’s harder to let everyone do their own thing)
tunability-of-AGI-power / implementability of the harm principle (if you can give everyone AGI that can follow very well the rule “don’t let these people harm other people”, then you can give that AGI safely to everyone and they can build planets however they like but not death ray anyone else’s planets)
The latter might be more of a “singleton that allows playgrounds” rather an actual multipolar world though.
Some of my general worries with singleton worlds are:
humanity has all its eggs in one basket—you better hope the governance structure is never corrupted, or never becomes sclerotic; real-life institutions so far have not given me many signs of hope on this count
cultural evolution is a pretty big part of how human societies seem to have improved and relies on a population of cultures / polities
vague instincts towards diversity being good and less fragile than homogeneity or centralisation
So to summarise: if we have a multipolar world, and the vulnerable world hypothesis if true, then conflict can be existentially bad and this is a reason to avoid a multipolar world. Didn’t consider this, interesting point!
(I also commented on substack)
This applies, but weaker even in a non-vulnerable world, because the incentives are way weaker for peaceful cooperation of values in AGI-world.
Considerations:
offense/defense balance (if offense wins very hard, it’s harder to let everyone do their own thing)
tunability-of-AGI-power / implementability of the harm principle (if you can give everyone AGI that can follow very well the rule “don’t let these people harm other people”, then you can give that AGI safely to everyone and they can build planets however they like but not death ray anyone else’s planets)
I do think this requires severely restraining open-source, but conditional on that happening, I think the offense-defense balance/tunability will sort of work out.
Some of my general worries with singleton worlds are:
humanity has all its eggs in one basket—you better hope the governance structure is never corrupted, or never becomes sclerotic; real-life institutions so far have not given me many signs of hope on this count
cultural evolution is a pretty big part of how human societies seem to have improved and relies on a population of cultures / polities
vague instincts towards diversity being good and less fragile than homogeneity or centralisation
Yeah, I’m not a fan of singleton worlds, and tend towards multipolar worlds. It’s just that it might involve a loss of a lot of life in the power-struggles around AGI.
On governing the commons, I’d say Elinor Ostrom’s observations are derivable from the folk theorems of game theory, which basically says that any outcome can be a Nash Equilibrium (with a few conditions that depend on the theorem) can be possible if the game is repeated and players have to deal with each other.
The problem is that AGI weakens the incentives for players to deal with each other, so Elinor Ostrom’s solutions are much less effective.
I believe that the near future (next 10 years) involves a fragile world and heavily offense-dominant tech, such that a cohesive governing body (not necessarily a single mind, it could be a coalition of multiple AIs and humans) will be necessary to enforce safety. Particularly, preventing the creation/deployment of self-replicating harms (rogue amoral AI, bioweapons, etc.).
On the other hand, I don’t think we can be sure what the more distant future (>50 years?) will look like. It may be that d/acc succeeds in advancing defense-dominant technology enough to make society more robust to violent defection. In such a world, it would be safe to have more multi-polar governance.
I am quite uncertain about how the world might transition to uni-polar governance, whether this will involve a singleton AI or a world government or a coalition of powerful AIs or what. Just that the ‘suicide switch’ for all of humanity and its AIs will for a time be quite cheap and accessible, and require quite a bit of surveillance and enforcement to ensure no defector can choose it.
My main default prediction here is that we will avoid either the absolute best case or the absolute worst case scenario, because I predict intent alignment works well enough to avoid extinction of humanity type scenarios, but I also don’t believe we will see radical movements toward equality (indeed the politics of our era is moving towards greater acceptance of inequality), so capitalism more or less survives the transition to AGI.
I do think dynamism will still exist, but it will be very limited to the upper classes/very rich of society, and most people will not be a part of it, and I’m including uploaded humans here in this calculation.
To address this:
To be somewhat more fair, the worry here is that in a regime where you don’t need society anymore because AIs can do all the work for your society, value conflicts become a bigger deal than today, because there is less reason to tolerate other people’s values if you can just found your own society based on your own values, and if you believe in the vulnerable world hypothesis, as a lot of rationalists do, then conflict has existential stakes, and even if not, can be quite bad, so one group controlling the future is better than inevitable conflict.
At a foundational level, whether or not our current tolerance for differing values is stable ultimately comes down to we can compensate for the effect of AGI allowing people to make their own society.
Comment is also on substack:
https://nosetgauge.substack.com/p/capital-agi-and-human-ambition/comment/83401326
How certain are you of this, and how much do you think it comes down more to something like “to what extent can disempowered groups unionise against the elite?”.
To be clear, by default I think AI will make unionising against the more powerful harder, but it might depend on the governance structure. Maybe if we are really careful, we can get something closer to “Direct Democracy”, where individual preferences actually matter more!
I am focused here on short-term politics in the US, which ordinarily would matter less, if it wasn’t likely that world-changing AI would be built in the US, but given that it might, it becomes way more important than normal.
So to summarise: if we have a multipolar world, and the vulnerable world hypothesis if true, then conflict can be existentially bad and this is a reason to avoid a multipolar world. Didn’t consider this, interesting point!
Considerations:
offense/defense balance (if offense wins very hard, it’s harder to let everyone do their own thing)
tunability-of-AGI-power / implementability of the harm principle (if you can give everyone AGI that can follow very well the rule “don’t let these people harm other people”, then you can give that AGI safely to everyone and they can build planets however they like but not death ray anyone else’s planets)
The latter might be more of a “singleton that allows playgrounds” rather an actual multipolar world though.
Some of my general worries with singleton worlds are:
humanity has all its eggs in one basket—you better hope the governance structure is never corrupted, or never becomes sclerotic; real-life institutions so far have not given me many signs of hope on this count
cultural evolution is a pretty big part of how human societies seem to have improved and relies on a population of cultures / polities
vague instincts towards diversity being good and less fragile than homogeneity or centralisation
Thanks!
(I also commented on substack)
This applies, but weaker even in a non-vulnerable world, because the incentives are way weaker for peaceful cooperation of values in AGI-world.
I do think this requires severely restraining open-source, but conditional on that happening, I think the offense-defense balance/tunability will sort of work out.
Yeah, I’m not a fan of singleton worlds, and tend towards multipolar worlds. It’s just that it might involve a loss of a lot of life in the power-struggles around AGI.
On governing the commons, I’d say Elinor Ostrom’s observations are derivable from the folk theorems of game theory, which basically says that any outcome can be a Nash Equilibrium (with a few conditions that depend on the theorem) can be possible if the game is repeated and players have to deal with each other.
The problem is that AGI weakens the incentives for players to deal with each other, so Elinor Ostrom’s solutions are much less effective.
More here:
https://en.wikipedia.org/wiki/Folk_theorem_(game_theory)
I believe that the near future (next 10 years) involves a fragile world and heavily offense-dominant tech, such that a cohesive governing body (not necessarily a single mind, it could be a coalition of multiple AIs and humans) will be necessary to enforce safety. Particularly, preventing the creation/deployment of self-replicating harms (rogue amoral AI, bioweapons, etc.).
On the other hand, I don’t think we can be sure what the more distant future (>50 years?) will look like. It may be that d/acc succeeds in advancing defense-dominant technology enough to make society more robust to violent defection. In such a world, it would be safe to have more multi-polar governance.
I am quite uncertain about how the world might transition to uni-polar governance, whether this will involve a singleton AI or a world government or a coalition of powerful AIs or what. Just that the ‘suicide switch’ for all of humanity and its AIs will for a time be quite cheap and accessible, and require quite a bit of surveillance and enforcement to ensure no defector can choose it.