But I don’t think anyone’s extrapolated volition would be to build their utopias in the real world. Post-ASI, virtual is strictly better. No one wants his utopia constrained by the laws of physics.
And it seems unlikely that anyone would choose to spend extended periods of time with pre-ASI humans rather than people made bespoke for them.
Also, it’s not clear to me that we will get a bargaining scenario. Aligned ASI could just impose equal apportioning of compute budget. This depends on how AI progress plays out.
Great stuff.
But I don’t think anyone’s extrapolated volition would be to build their utopias in the real world. Post-ASI, virtual is strictly better. No one wants his utopia constrained by the laws of physics.
And it seems unlikely that anyone would choose to spend extended periods of time with pre-ASI humans rather than people made bespoke for them.
Also, it’s not clear to me that we will get a bargaining scenario. Aligned ASI could just impose equal apportioning of compute budget. This depends on how AI progress plays out.
Well. Maybe.