I think “there is a lot of possible misaligned ASI, you can’t guess them all” is pretty much valid argument? If space of all Earth-originated misaligned superintelligences is described by 100 bits, therefore you need 2^100 ~ 10^33 simulations and pay 10^34 planets, which, given the fact that observable universe has ~10^80 protons in it and Earth has ~10^50 atoms, is beyond our ability to pay. If you pay the entire universe by doing 10^29 simulations, any misaligned ASI will consider probability of being in simulation to be 0.0001 and obviously take 1 planet over 0.001 expected.
I think the acausal trade framework rest on the assumption that we are in a (quantum or Tegmark) multiverse. Then, it’s not one human civilization in one branch that needs to do all the 2^100 trades: we just spin a big quantum wheel, and trade with the AI that comes up. (that’s why I wrote “humans can relatively accurately sample from the distribution of possible human-created unaligned AI values”). Thus, every AI will get a trade partner in some branch, and altogether the math checks out. Every AI has around 2^{-100} measure in base realities, and gets traded with in 2^{-100} portion of the human-controlled worlds, and the humans offer more planets than what they ask for, so it’s a good deal for the AI.
If you don’t buy the mutiverse premise (which is fair), then I think you shouldn’t think in terms of acausal trade in the first place, but consider my original proposal with simulations. I don’t see how the diversity of AI values is a problem there, the only important thing is that the AI should believe that it’s more likely than not to be in a human-run simulation.
Okay, I defer to you that the different possible worlds in the prior don’t need to “actually exist” for the acausal trade to go through. However, do I still understand correctly that spinning the quantum wheel should just work, and it’s not one branch of human civilization that needs to simulate all the possible AIs, right?
However, do I still understand correctly that spinning the quantum wheel should just work, and it’s not one branch of human civilization that needs to simulate all the possible AIs, right?
we just spin a big quantum wheel, and trade with the AI that comes up
Or run a computation to approximate an average, if that’s possible.
I’d guess it must be possible if you can randomly sample, at least. I.e., if you mean sampling from some set of worlds, and not just randomly combinatorially generating programs until you find a trade partner.
I think “there is a lot of possible misaligned ASI, you can’t guess them all” is pretty much valid argument? If space of all Earth-originated misaligned superintelligences is described by 100 bits, therefore you need 2^100 ~ 10^33 simulations and pay 10^34 planets, which, given the fact that observable universe has ~10^80 protons in it and Earth has ~10^50 atoms, is beyond our ability to pay. If you pay the entire universe by doing 10^29 simulations, any misaligned ASI will consider probability of being in simulation to be 0.0001 and obviously take 1 planet over 0.001 expected.
I think the acausal trade framework rest on the assumption that we are in a (quantum or Tegmark) multiverse. Then, it’s not one human civilization in one branch that needs to do all the 2^100 trades: we just spin a big quantum wheel, and trade with the AI that comes up. (that’s why I wrote “humans can relatively accurately sample from the distribution of possible human-created unaligned AI values”). Thus, every AI will get a trade partner in some branch, and altogether the math checks out. Every AI has around 2^{-100} measure in base realities, and gets traded with in 2^{-100} portion of the human-controlled worlds, and the humans offer more planets than what they ask for, so it’s a good deal for the AI.
If you don’t buy the mutiverse premise (which is fair), then I think you shouldn’t think in terms of acausal trade in the first place, but consider my original proposal with simulations. I don’t see how the diversity of AI values is a problem there, the only important thing is that the AI should believe that it’s more likely than not to be in a human-run simulation.
I think the argument should also go through without simulations and without the multiverse so long as you are a UDT-ish agent with a reasonable prior.
Okay, I defer to you that the different possible worlds in the prior don’t need to “actually exist” for the acausal trade to go through. However, do I still understand correctly that spinning the quantum wheel should just work, and it’s not one branch of human civilization that needs to simulate all the possible AIs, right?
This is my understanding.
Or run a computation to approximate an average, if that’s possible.
I’d guess it must be possible if you can randomly sample, at least. I.e., if you mean sampling from some set of worlds, and not just randomly combinatorially generating programs until you find a trade partner.