I’m not figuring it out enough to fully clarify, but: I feel there’s some sort of analysis missing here, which would clarify some of the main questions. Something around: What sorts of things can you actually bargain/negotiate/trade for, when the only thing that matters is differences of value? (As opposed to differences of capability.)
On the one hand, you have some severe “nonlinearities” (<-metaphor, I think? really I mean “changes in behavior-space that don’t trade off very strongly between different values”).
E.g. we might ask the AI: hey, you are running simulations of the humans you took Earth from. You’re torturing them horribly for thousands of years. But look, you can tweak your sims, and you get almost as much of the info you wanted, but now there’s no suffering. Please do this (at very low cost to you, great benefit to us) and we’ll give you a planet (low cost to us, low benefit to you).
On the other hand, you have direct tradeoffs.
E.g., everybody needs a Thneed. You have a Thneed. You could give it to me, but that would cost you 1 Thneed and gain me 1 Thneed. This is of negative value (transaction costs). E.g. energy, matter, etc.
“Just leave them the solar system” is asking for a trade of Thneeds. Everybody wants to eat Earth.
If humane civilization gets 10% of (some subset, starting from some earlier checkpoint, of...?) the lightcone, then they can bargain for at most 10% of other Earths to survive, right? And probably a lot less.
This seems to lead to the repugnant conclusion, where humanity is 80% dead or worse; 10% meager existence on a caged Earth; and 10% custodians of a vast array of AIs presiding over solar systems.
I don’t understand why only 10% of Earths could survive if humanity only gets 10% of the Lightcone in expectation. Like the whole point is that we (or at least personally, I) want to keep Earth much more than how much most AIs want to eat it. So we can trade 10 far-away extra planets in the worlds we win, for keeping Earth in the worlds we lose. If we get an AI who is not a universal paperclip maximizer and deeply cares about doing things with Earth in particular (maybe that’s what you mean by Thneed? I don’t understand what that is), then I agree that’s rough, and it falls under the objection that I acknowledge, that there might be AIs with whom we can’t find a compromise, but I expect this to be relatively rare.
Nevermind, I was confused, my bad. Yeah you can save a lot more than 10% of the Earths.
As a separate point, I do worry that some other nonhumane coalition has vastly more bargaining power compared to the humane one, by virtue of happening 10 million years ago or whatever. In this case, AIs would tend to realize this fact, and then commit-before-simulation-aware to “figure out what the dominant coalition wants to trade about”.
They got way more of the Everett branches, so to speak. Suppose that the Pseudosuchians had a 20% chance of producing croc-FAI. So starting at the Triassic, you have that 20% of worlds become croc-god worlds, and 80% become a mix of X-god worlds for very many different Xs; maybe only 5% of worlds produce humans, and only .01% produce Humane-gods.
Maybe doing this with Pseudosuchians is less plausible than with humans because you can more easily model what Humane-gods would bargain for, because you have access to humans. But that’s eyebrow-raising. What about Corvid-gods, etc. If you can do more work and get access to vastly more powerful acausal trade partners, seems worth it; and, on the face of it, the leap from [acausal trade is infeasible, period] to [actually acausal trade with hypothetical Humane-gods is feasible] seems bigger than the jump from [trade with Humane-gods is feasible] to [trade with Corvid-gods is feasible] or [trade with Cetacean-gods is feasible], though IDK of course. (Then there’s the jump to [trade with arbitrary gods from the multiverse]. IDK.)
I’m not figuring it out enough to fully clarify, but: I feel there’s some sort of analysis missing here, which would clarify some of the main questions. Something around: What sorts of things can you actually bargain/negotiate/trade for, when the only thing that matters is differences of value? (As opposed to differences of capability.)
On the one hand, you have some severe “nonlinearities” (<-metaphor, I think? really I mean “changes in behavior-space that don’t trade off very strongly between different values”).
E.g. we might ask the AI: hey, you are running simulations of the humans you took Earth from. You’re torturing them horribly for thousands of years. But look, you can tweak your sims, and you get almost as much of the info you wanted, but now there’s no suffering. Please do this (at very low cost to you, great benefit to us) and we’ll give you a planet (low cost to us, low benefit to you).
On the other hand, you have direct tradeoffs.
E.g., everybody needs a Thneed. You have a Thneed. You could give it to me, but that would cost you 1 Thneed and gain me 1 Thneed. This is of negative value (transaction costs). E.g. energy, matter, etc.
“Just leave them the solar system” is asking for a trade of Thneeds. Everybody wants to eat Earth.
If humane civilization gets 10% of (some subset, starting from some earlier checkpoint, of...?) the lightcone, then they can bargain for at most 10% of other Earths to survive, right? And probably a lot less.
This seems to lead to the repugnant conclusion, where humanity is 80% dead or worse; 10% meager existence on a caged Earth; and 10% custodians of a vast array of AIs presiding over solar systems.
I don’t understand why only 10% of Earths could survive if humanity only gets 10% of the Lightcone in expectation. Like the whole point is that we (or at least personally, I) want to keep Earth much more than how much most AIs want to eat it. So we can trade 10 far-away extra planets in the worlds we win, for keeping Earth in the worlds we lose. If we get an AI who is not a universal paperclip maximizer and deeply cares about doing things with Earth in particular (maybe that’s what you mean by Thneed? I don’t understand what that is), then I agree that’s rough, and it falls under the objection that I acknowledge, that there might be AIs with whom we can’t find a compromise, but I expect this to be relatively rare.
Nevermind, I was confused, my bad. Yeah you can save a lot more than 10% of the Earths.
As a separate point, I do worry that some other nonhumane coalition has vastly more bargaining power compared to the humane one, by virtue of happening 10 million years ago or whatever. In this case, AIs would tend to realize this fact, and then commit-before-simulation-aware to “figure out what the dominant coalition wants to trade about”.
Why would the time it happens at matter?
They got way more of the Everett branches, so to speak. Suppose that the Pseudosuchians had a 20% chance of producing croc-FAI. So starting at the Triassic, you have that 20% of worlds become croc-god worlds, and 80% become a mix of X-god worlds for very many different Xs; maybe only 5% of worlds produce humans, and only .01% produce Humane-gods.
Maybe doing this with Pseudosuchians is less plausible than with humans because you can more easily model what Humane-gods would bargain for, because you have access to humans. But that’s eyebrow-raising. What about Corvid-gods, etc. If you can do more work and get access to vastly more powerful acausal trade partners, seems worth it; and, on the face of it, the leap from [acausal trade is infeasible, period] to [actually acausal trade with hypothetical Humane-gods is feasible] seems bigger than the jump from [trade with Humane-gods is feasible] to [trade with Corvid-gods is feasible] or [trade with Cetacean-gods is feasible], though IDK of course. (Then there’s the jump to [trade with arbitrary gods from the multiverse]. IDK.)