With randomization, you reduce the cost and the upside in concert. If a pair of shoes costs $100, and that’s more than I’m willing to pay, I could buy the shoes with probability 1%, and it will only cost me $1 in expectation, but I will only get the shoes with probability 1⁄100.
I agree that randomization reduces the “upside” in the sense of “reducing our weight in the universal prior.” But utility is not linear in that weight.
I’m saying that the consequentialists completely dominate the universal prior, and they will still completely dominate if you reduce their weight by 2x. So either way they get all the influence. (Quantitatively, suppose the consequentialists currently have probability 1000 times greater than the intended model. Then they have 99.9% of the posterior. If they decreased their probability of acting by two, then they’d have 500 times the probability of the intended model, and so have 99.8% of the posterior. This is almost as good as 99.9%.)
That could fail if e.g. if there are a bunch of other consequentialists also trying to control the sequence. Or if some other model beyond the intended one has much higher probability. But if you think that the consequentialists are X bits simpler than the intended model, and you are trying to argue that the intended model dominates the posterior, then you need to argue that the consequentialists wouldn’t try to grab the universal prior even when doing so only requires acting in 2−X of worlds.
Someone in the basement universe is reasoning about the output of a randomized Turing machine that I’m running on.
I care about what they believe about that Turing machine. Namely, I want them to believe that most of the time when the sequence x appears, it is followed by a 1.
Their beliefs depend in a linear way on my probabilities of action.
(At least if e.g. I committed to that policy at an early enough time for them to reason about it, or if my policy is sufficiently predictable to be correlated with their predictions, or if they are able to actually simulate me in a universe with reflective oracles… If I’m not able to influence their beliefs about me, then of course I can’t influence their beliefs about anything and the whole manipulative project doesn’t get off the ground.)
But my utility is a non-linear function of their beliefs, since P(1|x) is a non-linear function of their beliefs.
So my utility is a non-linear function of my policy.
With randomization, you reduce the cost and the upside in concert. If a pair of shoes costs $100, and that’s more than I’m willing to pay, I could buy the shoes with probability 1%, and it will only cost me $1 in expectation, but I will only get the shoes with probability 1⁄100.
I agree that randomization reduces the “upside” in the sense of “reducing our weight in the universal prior.” But utility is not linear in that weight.
I’m saying that the consequentialists completely dominate the universal prior, and they will still completely dominate if you reduce their weight by 2x. So either way they get all the influence. (Quantitatively, suppose the consequentialists currently have probability 1000 times greater than the intended model. Then they have 99.9% of the posterior. If they decreased their probability of acting by two, then they’d have 500 times the probability of the intended model, and so have 99.8% of the posterior. This is almost as good as 99.9%.)
That could fail if e.g. if there are a bunch of other consequentialists also trying to control the sequence. Or if some other model beyond the intended one has much higher probability. But if you think that the consequentialists are X bits simpler than the intended model, and you are trying to argue that the intended model dominates the posterior, then you need to argue that the consequentialists wouldn’t try to grab the universal prior even when doing so only requires acting in 2−X of worlds.
If I flip a coin to randomize between two policies, I don’t see how that mixed policy could produce more value for me than the base policies.
(ETA: the logical implications about the fact of my randomization don’t have any weird anti-adversarial effects here).
Someone in the basement universe is reasoning about the output of a randomized Turing machine that I’m running on.
I care about what they believe about that Turing machine. Namely, I want them to believe that most of the time when the sequence x appears, it is followed by a 1.
Their beliefs depend in a linear way on my probabilities of action.
(At least if e.g. I committed to that policy at an early enough time for them to reason about it, or if my policy is sufficiently predictable to be correlated with their predictions, or if they are able to actually simulate me in a universe with reflective oracles… If I’m not able to influence their beliefs about me, then of course I can’t influence their beliefs about anything and the whole manipulative project doesn’t get off the ground.)
But my utility is a non-linear function of their beliefs, since P(1|x) is a non-linear function of their beliefs.
So my utility is a non-linear function of my policy.