Things are confusing because there are lots of different dominance relations that people talk about. There’s a dominance relation on strategies, and there are (multiple) dominance relations on lotteries.
Here are the definitions I’m working with.
A strategy is a plan about which options to pick at each choice-node in a decision-tree.
Strategies yield lotteries (rather than final outcomes) when the plan involves passing through a chance-node. For example, consider the decision-tree below:
A strategy specifies what option the agent would pick at choice-node 1, what option the agent would pick at choice-node 2, and what option the agent would pick at choice-node 3.
Suppose that the agent’s strategy is {Pick B at choice-node 1, Pick A+ at choice-node 2, Pick B at choice-node 3}. This strategy doesn’t yield a final outcome, because the agent doesn’t get to decide what happens at the chance-node. Instead, the strategy yields the lottery 0.5(A+)+0.5(B). This just says that: if the agent executes the strategy, then there’s a 0.5 probability that they end up with final outcome A+ and a 0.5 probability that they end up with final outcome B.
The dominance relation on strategies has to refer to the lotteries yielded by strategies, rather than the final outcomes yielded by strategies, because strategies don’t yield final outcomes when the agent passes through a chance-node.[1] So we define the dominance relation on strategies as follows:
Strategy Dominance (relation)
A strategy S is dominated by a strategy S’ iff S yields a lottery X that is strictly dispreferred to the lottery X’ yielded by S’.
Now for the dominance relations on lotteries.[2] One is:
Statewise Dominance (relation)
Lottery X statewise-dominates lottery Y iff, in each state [environment_random_seed], X yields a final outcome weakly preferred to the final outcome yielded by Y, and in some state [environment_random_seed], X yields a final outcome strictly preferred to the final outcome yielded by Y.
Another is:
Statewise Pseudodominance (relation)
Lottery X statewise-pseudodominates lottery Y iff, in each state [environment_random_seed], X yields a final outcome weakly preferred to or pref-gappedto the final outcome yielded by Y, and in some state [environment_random_seed], X yields a final outcome strictly preferred to the final outcome yielded by Y.
The lottery A (that yields final outcome A for sure) is statewise-pseudodominated by the lottery 0.5(A+)+0.5(B), but it isn’tstatewise-dominated by 0.5(A+)+0.5(B). That’s because the agent has a preferential gap between the final outcomes A and B.
Advanced agents with incomplete preferences over final outcomes will plausibly satisfy the Statewise Dominance Principle:
Statewise Dominance Principle
If lottery X statewise-dominates lottery Y, then the agent strictly prefers X to Y.
And that’s because agents that violate the Statewise Dominance Principle are ‘shooting themselves in the foot’ in the relevant sense. If the agent executes a strategy that yields a statewise-dominated lottery, then there’s another available strategy that—in each state—gives a final outcome that is at least as good in every respect that the agent cares about, and—in some state—gives a final outcome that is better in some respect that the agent cares about.
But advanced agents with incomplete preferences over final outcomes plausibly won’t satisfy the Statewise Pseudodominance Principle:
Statewise Pseudodominance Principle
If lottery X statewise-pseudodominates lottery Y, then the agent strictly prefers X to Y.
And that’s for the reasons that I gave in my comment above. Condensing:
A statewise-pseudodominated lottery can be such that, in some state, that lottery is better than all other available lotteries in some respect that the agent cares about.
The statewise pseudodominance relation is cyclic, so the Statewise Pseudodominance Principle would lead to cyclic preferences.
The decision about whether to go to that chance node should be derived from the final outcomes, not from some newly created terminal preference about that chance node.
But:
- The decision can also depend on the probabilities of those final outcomes.
- The decision is constrained by preferences over final outcomes and probabilities of those final outcomes. I’m supposing that the agent’s preferences over lotteries depends only on these lotteries’ possible final outcomes and their probabilities. I’m not supposing that the agent has newly created terminal preferences/arbitrary preferences about non-final states.
There are stochastic versions of each of these relations, which ignore how states line up across lotteries and instead talk about probabilities of outcomes. I think everything I say below is also true for the stochastic versions.
[Edit: I think I misinterpreted EJT in a way that invalidates some of this comment, see downthread comment clarifying this].
That is really helpful, thanks. I had been making a mistake, in that I thought that there was an argument from just “the agent thinks it’s possible the agent will run into a money pump” that concluded “the agent should complete that preference in advance”. But I was thinking sloppily and accidentally sometimes equivocating between pref-gaps and indifference. So I don’t think this argument works by itself, but I think it might be made to work with an additional assumption.
One intuition that I find convincing is that if I found myself at outcome A in the single sweetening money pump, I would regret having not made it to A+. This intuition seems to hold even if I imagine A and B to be of incomparable value.
In order to avoid this regret, I would try to become the sort of agent that never found itself in that position. I can see that if I always follow the Caprice rule, then it’s a little weird to regret not getting A+, because that isn’t a counterfactually available option (counterfacting on decision 1). But this feels like I’m being cheated. I think the reason that if feels like I’m being cheated is that I feel like getting to A+ should be a counterfactually available option.
One way to make it a counterfactually available option in the thought experiment is to introduce another choice before choice 1 in the decision tree. The new choice (0), is the choice about whether to maintain the same decision algorithm (call this incomplete), or complete the preferential gap between A and B (call this complete).
I think the choice complete statewise dominates incomplete. This is because the choice incomplete results in a lottery {B: qp, A+: q(1−p), A:(1−q)} for q<1.[1] However, the choice complete results in the lottery {B: p, A+: (1−p), A:0}.
Do you disagree with this? I think this allows us to create a money pump, by charging the agent $ϵ for the option to complete its own preferences.
The statewise pseudodominance relation is cyclic, so the Statewise Pseudodominance Principle would lead to cyclic preferences.
This still seems wrong to me, because I see lotteries as being an object whose purpose is to summarize random variables and outcomes. So it’s weird to compare lotteries that depend on the same random variables (they are correlated), as if they are independent. This seems like a sidetrack though, and it’s plausible to me that I’m just confused about your definitions here.
Letting p be the probability that the agent chooses 2A+ and (1−p) the probability the agent chooses 2B (following your comment above). And q is defined similarly, for choice 1.
I made a mistake again. As described above, complete only pseudodominates incomplete.
But this is easily patched with the trick described in the OP. So we need the choice complete to make two changes to the downstream decisions. First, change decision 1 to always choose up (as before), second, change the distribution of Decision 2 to {1−q(1−p), q(1−p)}, because this keeps the probability of B constant. Fixed diagram:
Now the lottery for complete is {B: q(1−p), A+: 1−q(1−p), A:0}, and the lottery for incomplete is {B: q(1−p), A+: pq, A:(1−q)}. So overall, there is a pure shift of probability from A to A+. [Edit 23/7: hilariously, I still had the probabilities wrong, so fixed them, again].
I think the above money pump works, if the agent sometimes chooses the A path, but I was incorrect in thinking that the caprice rule sometimes chooses the A path.
I misinterpreted one of EJT’s comments as saying it might choose the A path. The last couple of days I’ve been reading through some of the sources he linked to in the original “there are no coherence theorems” post and one of them (Gustafsson) made me realize I was interpreting him incorrectly, by simplifying the decision tree in a way that doesn’t make sense. I only realized this yesterday.
Now I think that the caprice rule is essentially equivalent to updatelessness. If I understand correctly, it would be equivalent to 1. choosing the best policy by ranking them in the partial order of outcomes (randomizing over multiple maxima), then 2. implementing that policy without further consideration. And this makes it immune to money pumps and renders any self-modification pointless. It also makes it behaviorally indistinguishable from an agent with complete preferences, as far as I can tell. The same updatelessness trick seems to apply to all money pump arguments. It’s what scott uses in this post to avoid the independence money pump.
So currently I’m thinking updatelessness removes most of the justification for the VNM axioms (including transitivity!). But I’m confused because updateless policies still must satisfy local properties like “doesn’t waste resources unless it helps achieve the goal”, which is intuitively what the money pump arguments represent. So there must be some way to recover properties like this. Maybe via John’s approach here. But I’m only maybe 80% sure of my new understanding, I’m still trying to work through it all.
It looks to me like the “updatelessness trick” you describe (essentially, behaving as though certain non-local branches of the decision tree are still counterfactually relevant even though they are not — although note that I currently don’t see an obvious way to use that to avoid the usual money pump against intransitivity) recovers most of the behavior we’d see under VNM anyway; and so I don’t think I understand your confusion re: VNM axioms.
E.g. can you give me a case in which (a) we have an agent that exhibits preferences against whose naive implementation there exists some kind of money pump (not necessarily a repeatable one), (b) the agent can implement the updatelessness trick in order to avoid the money pump without modifying their preferences, and yet (c) the agent is not then representable as having modified their preferences in the relevant way?
Things are confusing because there are lots of different dominance relations that people talk about. There’s a dominance relation on strategies, and there are (multiple) dominance relations on lotteries.
Here are the definitions I’m working with.
A strategy is a plan about which options to pick at each choice-node in a decision-tree.
Strategies yield lotteries (rather than final outcomes) when the plan involves passing through a chance-node. For example, consider the decision-tree below:
A strategy specifies what option the agent would pick at choice-node 1, what option the agent would pick at choice-node 2, and what option the agent would pick at choice-node 3.
Suppose that the agent’s strategy is {Pick B at choice-node 1, Pick A+ at choice-node 2, Pick B at choice-node 3}. This strategy doesn’t yield a final outcome, because the agent doesn’t get to decide what happens at the chance-node. Instead, the strategy yields the lottery 0.5(A+)+0.5(B). This just says that: if the agent executes the strategy, then there’s a 0.5 probability that they end up with final outcome A+ and a 0.5 probability that they end up with final outcome B.
The dominance relation on strategies has to refer to the lotteries yielded by strategies, rather than the final outcomes yielded by strategies, because strategies don’t yield final outcomes when the agent passes through a chance-node.[1] So we define the dominance relation on strategies as follows:
Now for the dominance relations on lotteries.[2] One is:
Another is:
The lottery A (that yields final outcome A for sure) is statewise-pseudodominated by the lottery 0.5(A+)+0.5(B), but it isn’t statewise-dominated by 0.5(A+)+0.5(B). That’s because the agent has a preferential gap between the final outcomes A and B.
Advanced agents with incomplete preferences over final outcomes will plausibly satisfy the Statewise Dominance Principle:
And that’s because agents that violate the Statewise Dominance Principle are ‘shooting themselves in the foot’ in the relevant sense. If the agent executes a strategy that yields a statewise-dominated lottery, then there’s another available strategy that—in each state—gives a final outcome that is at least as good in every respect that the agent cares about, and—in some state—gives a final outcome that is better in some respect that the agent cares about.
But advanced agents with incomplete preferences over final outcomes plausibly won’t satisfy the Statewise Pseudodominance Principle:
And that’s for the reasons that I gave in my comment above. Condensing:
A statewise-pseudodominated lottery can be such that, in some state, that lottery is better than all other available lotteries in some respect that the agent cares about.
The statewise pseudodominance relation is cyclic, so the Statewise Pseudodominance Principle would lead to cyclic preferences.
You say:
But:
- The decision can also depend on the probabilities of those final outcomes.
- The decision is constrained by preferences over final outcomes and probabilities of those final outcomes. I’m supposing that the agent’s preferences over lotteries depends only on these lotteries’ possible final outcomes and their probabilities. I’m not supposing that the agent has newly created terminal preferences/arbitrary preferences about non-final states.
There are stochastic versions of each of these relations, which ignore how states line up across lotteries and instead talk about probabilities of outcomes. I think everything I say below is also true for the stochastic versions.
[Edit: I think I misinterpreted EJT in a way that invalidates some of this comment, see downthread comment clarifying this].
That is really helpful, thanks. I had been making a mistake, in that I thought that there was an argument from just “the agent thinks it’s possible the agent will run into a money pump” that concluded “the agent should complete that preference in advance”. But I was thinking sloppily and accidentally sometimes equivocating between pref-gaps and indifference. So I don’t think this argument works by itself, but I think it might be made to work with an additional assumption.
One intuition that I find convincing is that if I found myself at outcome A in the single sweetening money pump, I would regret having not made it to A+. This intuition seems to hold even if I imagine A and B to be of incomparable value.
In order to avoid this regret, I would try to become the sort of agent that never found itself in that position. I can see that if I always follow the Caprice rule, then it’s a little weird to regret not getting A+, because that isn’t a counterfactually available option (counterfacting on decision 1). But this feels like I’m being cheated. I think the reason that if feels like I’m being cheated is that I feel like getting to A+ should be a counterfactually available option.
One way to make it a counterfactually available option in the thought experiment is to introduce another choice before choice 1 in the decision tree. The new choice (0), is the choice about whether to maintain the same decision algorithm (call this incomplete), or complete the preferential gap between A and B (call this complete).
I think the choice complete statewise dominates incomplete. This is because the choice incomplete results in a lottery {B: qp, A+: q(1−p), A:(1−q)} for q<1.[1] However, the choice complete results in the lottery {B: p, A+: (1−p), A:0}.
Do you disagree with this? I think this allows us to create a money pump, by charging the agent $ϵ for the option to complete its own preferences.
This still seems wrong to me, because I see lotteries as being an object whose purpose is to summarize random variables and outcomes. So it’s weird to compare lotteries that depend on the same random variables (they are correlated), as if they are independent. This seems like a sidetrack though, and it’s plausible to me that I’m just confused about your definitions here.
Letting p be the probability that the agent chooses 2A+ and (1−p) the probability the agent chooses 2B (following your comment above). And q is defined similarly, for choice 1.
I made a mistake again. As described above, complete only pseudodominates incomplete.
But this is easily patched with the trick described in the OP. So we need the choice complete to make two changes to the downstream decisions. First, change decision 1 to always choose up (as before), second, change the distribution of Decision 2 to {1−q(1−p), q(1−p)}, because this keeps the probability of B constant. Fixed diagram:
Now the lottery for complete is {B: q(1−p), A+: 1−q(1−p), A:0}, and the lottery for incomplete is {B: q(1−p), A+: pq, A:(1−q)}. So overall, there is a pure shift of probability from A to A+.
[Edit 23/7: hilariously, I still had the probabilities wrong, so fixed them, again].
I think the above money pump works, if the agent sometimes chooses the A path, but I was incorrect in thinking that the caprice rule sometimes chooses the A path.
I misinterpreted one of EJT’s comments as saying it might choose the A path. The last couple of days I’ve been reading through some of the sources he linked to in the original “there are no coherence theorems” post and one of them (Gustafsson) made me realize I was interpreting him incorrectly, by simplifying the decision tree in a way that doesn’t make sense. I only realized this yesterday.
Now I think that the caprice rule is essentially equivalent to updatelessness. If I understand correctly, it would be equivalent to 1. choosing the best policy by ranking them in the partial order of outcomes (randomizing over multiple maxima), then 2. implementing that policy without further consideration. And this makes it immune to money pumps and renders any self-modification pointless. It also makes it behaviorally indistinguishable from an agent with complete preferences, as far as I can tell.
The same updatelessness trick seems to apply to all money pump arguments. It’s what scott uses in this post to avoid the independence money pump.
So currently I’m thinking updatelessness removes most of the justification for the VNM axioms (including transitivity!). But I’m confused because updateless policies still must satisfy local properties like “doesn’t waste resources unless it helps achieve the goal”, which is intuitively what the money pump arguments represent. So there must be some way to recover properties like this. Maybe via John’s approach here.
But I’m only maybe 80% sure of my new understanding, I’m still trying to work through it all.
It looks to me like the “updatelessness trick” you describe (essentially, behaving as though certain non-local branches of the decision tree are still counterfactually relevant even though they are not — although note that I currently don’t see an obvious way to use that to avoid the usual money pump against intransitivity) recovers most of the behavior we’d see under VNM anyway; and so I don’t think I understand your confusion re: VNM axioms.
E.g. can you give me a case in which (a) we have an agent that exhibits preferences against whose naive implementation there exists some kind of money pump (not necessarily a repeatable one), (b) the agent can implement the updatelessness trick in order to avoid the money pump without modifying their preferences, and yet (c) the agent is not then representable as having modified their preferences in the relevant way?