I think “there is a lot of possible misaligned ASI, you can’t guess them all” is pretty much valid argument? If space of all Earth-originated misaligned superintelligences is described by 100 bits, therefore you need 2^100 ~ 10^33 simulations and pay 10^34 planets, which, given the fact that observable universe has ~10^80 protons in it and Earth has ~10^50 atoms, is beyond our ability to pay. If you pay the entire universe by doing 10^29 simulations, any misaligned ASI will consider probability of being in simulation to be 0.0001 and obviously take 1 planet over 0.001 expected.
I think the acausal trade framework rest on the assumption that we are in a (quantum or Tegmark) multiverse. Then, it’s not one human civilization in one branch that needs to do all the 2^100 trades: we just spin a big quantum wheel, and trade with the AI that comes up. (that’s why I wrote “humans can relatively accurately sample from the distribution of possible human-created unaligned AI values”). Thus, every AI will get a trade partner in some branch, and altogether the math checks out. Every AI has around 2^{-100} measure in base realities, and gets traded with in 2^{-100} portion of the human-controlled worlds, and the humans offer more planets than what they ask for, so it’s a good deal for the AI.
If you don’t buy the mutiverse premise (which is fair), then I think you shouldn’t think in terms of acausal trade in the first place, but consider my original proposal with simulations. I don’t see how the diversity of AI values is a problem there, the only important thing is that the AI should believe that it’s more likely than not to be in a human-run simulation.
I think “there is a lot of possible misaligned ASI, you can’t guess them all” is pretty much valid argument? If space of all Earth-originated misaligned superintelligences is described by 100 bits, therefore you need 2^100 ~ 10^33 simulations and pay 10^34 planets, which, given the fact that observable universe has ~10^80 protons in it and Earth has ~10^50 atoms, is beyond our ability to pay. If you pay the entire universe by doing 10^29 simulations, any misaligned ASI will consider probability of being in simulation to be 0.0001 and obviously take 1 planet over 0.001 expected.
I think the acausal trade framework rest on the assumption that we are in a (quantum or Tegmark) multiverse. Then, it’s not one human civilization in one branch that needs to do all the 2^100 trades: we just spin a big quantum wheel, and trade with the AI that comes up. (that’s why I wrote “humans can relatively accurately sample from the distribution of possible human-created unaligned AI values”). Thus, every AI will get a trade partner in some branch, and altogether the math checks out. Every AI has around 2^{-100} measure in base realities, and gets traded with in 2^{-100} portion of the human-controlled worlds, and the humans offer more planets than what they ask for, so it’s a good deal for the AI.
If you don’t buy the mutiverse premise (which is fair), then I think you shouldn’t think in terms of acausal trade in the first place, but consider my original proposal with simulations. I don’t see how the diversity of AI values is a problem there, the only important thing is that the AI should believe that it’s more likely than not to be in a human-run simulation.