I think the above money pump works, if the agent sometimes chooses the A path, but I was incorrect in thinking that the caprice rule sometimes chooses the A path.
I misinterpreted one of EJT’s comments as saying it might choose the A path. The last couple of days I’ve been reading through some of the sources he linked to in the original “there are no coherence theorems” post and one of them (Gustafsson) made me realize I was interpreting him incorrectly, by simplifying the decision tree in a way that doesn’t make sense. I only realized this yesterday.
Now I think that the caprice rule is essentially equivalent to updatelessness. If I understand correctly, it would be equivalent to 1. choosing the best policy by ranking them in the partial order of outcomes (randomizing over multiple maxima), then 2. implementing that policy without further consideration. And this makes it immune to money pumps and renders any self-modification pointless. It also makes it behaviorally indistinguishable from an agent with complete preferences, as far as I can tell. The same updatelessness trick seems to apply to all money pump arguments. It’s what scott uses in this post to avoid the independence money pump.
So currently I’m thinking updatelessness removes most of the justification for the VNM axioms (including transitivity!). But I’m confused because updateless policies still must satisfy local properties like “doesn’t waste resources unless it helps achieve the goal”, which is intuitively what the money pump arguments represent. So there must be some way to recover properties like this. Maybe via John’s approach here. But I’m only maybe 80% sure of my new understanding, I’m still trying to work through it all.
It looks to me like the “updatelessness trick” you describe (essentially, behaving as though certain non-local branches of the decision tree are still counterfactually relevant even though they are not — although note that I currently don’t see an obvious way to use that to avoid the usual money pump against intransitivity) recovers most of the behavior we’d see under VNM anyway; and so I don’t think I understand your confusion re: VNM axioms.
E.g. can you give me a case in which (a) we have an agent that exhibits preferences against whose naive implementation there exists some kind of money pump (not necessarily a repeatable one), (b) the agent can implement the updatelessness trick in order to avoid the money pump without modifying their preferences, and yet (c) the agent is not then representable as having modified their preferences in the relevant way?
What I meant by updatelessness removes most of the justification is the reason given here at the very beginning of “Against Resolute Choice”. In order to make a money pump that leads the agent in a circle, the agent has to continue accepting trades around a full preference loop. But if it has decided on the entire plan beforehand, it will just do any plan that involves <1 trip around the preference loop. (Although it’s unclear how it would settle on such a plan, maybe just stopping its search after a given time). It won’t (I think?) choose any plan that does multiple loops, because they are strictly worse.
After choosing this plan though, I think it is representable as VNM rational, as you say. And I’m not sure what to do with this. It does seem important.
However, I think Scott’s argument here satisfies (a) (b) and (c). I think the independence axiom might be special in this respect, because the money pump for independence is exploiting an update on new information.
I don’t think agents that avoid the money pump for cyclicity are representable as satisfying VNM, at least holding fixed the objects of preference (as we should). Resolute choosers with cyclic preferences will reliably choose B over A- at node 3, but they’ll reliably choose A- over B if choosing between these options ex nihilo. That’s not VNM representable, because it requires that the utility of A- be greater than the utility of B and. that the utility of B be greater than the utility of A-
Perhaps I’m misusing the word “representable”? But what I meant was that any single sequence of actions generate by the agent could also have been generated by an outcome-utility maximizer (that has the same world model). This seems like the relevant definition, right?
It also makes it behaviorally indistinguishable from an agent with complete preferences, as far as I can tell.
That’s not right. As I say in another comment:
And an agent abiding by the Caprice Rule can’t be represented as maximising utility, because its preferences are incomplete. In cases where the available trades aren’t arranged in some way that constitutes a money-pump, the agent can prefer (/reliably choose) A+ over A, and yet lack any preference between (/stochastically choose between) A+ and B, and lack any preference between (/stochastically choose between) A and B. Those patterns of preference/behaviour are allowed by the Caprice Rule.
Or consider another example. The agent trades A for B, then B for A, then declines to trade A for B+. That’s compatible with the Caprice rule, but not with complete preferences.
Or consider the pattern of behaviour that (I elsewhere argue) can make agents with incomplete preferences shutdownable. Agents abiding by the Caprice rule can refuse to pay costs to shift probability mass between A and B, and refuse to pay costs to shift probability mass between A and B+. Agents with complete preferences can’t do that.
The same updatelessness trick seems to apply to all money pump arguments.
[I’m going to use the phrase ‘resolute choice’ rather than ‘updatelessness.’ That seems like a more informative and less misleading description of the relevant phenomenon: making a plan and sticking to it. You can stick to a plan even if you update your beliefs. Also, in the posts on UDT, ‘updatelessness’ seems to refer to something importantly distinct from just making a plan and sticking to it.]
That’s right, but the drawbacks of resolute choice depend on the money pump to which you apply it. As Gustafsson notes, if an agent uses resolute choice to avoid the money pump for cyclic preferences, that agent has to choose against their strict preferences at some point. For example, they have to choose B at node 3 in the money pump below, even though—were they facing that choice ex nihilo—they’d prefer to choose A-.
There’s no such drawback for agents with incomplete preferences using resolute choice. As I note in this post, agents with incomplete preferences using resolute choice need never choose against their strict preferences. The agent’s past plan only has to serve as a tiebreaker: forcing a particular choice between options between which they’d otherwise lack a preference. For example, they have to choose B at node 2 in the money pump below. Were they facing that choice ex nihilo, they’d lack a preference between B and A-.
Are you saying that my description (following) is incorrect?
[incomplete preferences w/ caprice] would be equivalent to 1. choosing the best policy by ranking them in the partial order of outcomes (randomizing over multiple maxima), then 2. implementing that policy without further consideration.
Or are you saying that it is correct, but you disagree that this implies that it is “behaviorally indistinguishable from an agent with complete preferences”? If this is the case, then I think we might disagree on the definition of “behaviorally indistinguishable”? I’m using it like: If you observe a single sequence of actions from this agent (and knowing the agent’s world model), can you construct a utility function over outcomes that could have produced that sequence.
Or consider another example. The agent trades A for B, then B for A, then declines to trade A for B+. That’s compatible with the Caprice rule, but not with complete preferences.
This is compatible with a resolute outcome-utility maximizer (for whom A is a maxima). There’s no rule that says an agent must take the shortest route to the same outcome (right?).
As Gustafsson notes, if an agent uses resolute choice to avoid the money pump for cyclic preferences, that agent has to choose against their strict preferences at some point. ... There’s no such drawback for agents with incomplete preferences using resolute choice.
Sure, but why is that a drawback? It can’t be money pumped, right? Agents following resolute choice often choose against their local strict preferences in other decision problems. (E.g. Newcomb’s). And this is considered an argument in favour of resolute choice.
I think the above money pump works, if the agent sometimes chooses the A path, but I was incorrect in thinking that the caprice rule sometimes chooses the A path.
I misinterpreted one of EJT’s comments as saying it might choose the A path. The last couple of days I’ve been reading through some of the sources he linked to in the original “there are no coherence theorems” post and one of them (Gustafsson) made me realize I was interpreting him incorrectly, by simplifying the decision tree in a way that doesn’t make sense. I only realized this yesterday.
Now I think that the caprice rule is essentially equivalent to updatelessness. If I understand correctly, it would be equivalent to 1. choosing the best policy by ranking them in the partial order of outcomes (randomizing over multiple maxima), then 2. implementing that policy without further consideration. And this makes it immune to money pumps and renders any self-modification pointless. It also makes it behaviorally indistinguishable from an agent with complete preferences, as far as I can tell.
The same updatelessness trick seems to apply to all money pump arguments. It’s what scott uses in this post to avoid the independence money pump.
So currently I’m thinking updatelessness removes most of the justification for the VNM axioms (including transitivity!). But I’m confused because updateless policies still must satisfy local properties like “doesn’t waste resources unless it helps achieve the goal”, which is intuitively what the money pump arguments represent. So there must be some way to recover properties like this. Maybe via John’s approach here.
But I’m only maybe 80% sure of my new understanding, I’m still trying to work through it all.
It looks to me like the “updatelessness trick” you describe (essentially, behaving as though certain non-local branches of the decision tree are still counterfactually relevant even though they are not — although note that I currently don’t see an obvious way to use that to avoid the usual money pump against intransitivity) recovers most of the behavior we’d see under VNM anyway; and so I don’t think I understand your confusion re: VNM axioms.
E.g. can you give me a case in which (a) we have an agent that exhibits preferences against whose naive implementation there exists some kind of money pump (not necessarily a repeatable one), (b) the agent can implement the updatelessness trick in order to avoid the money pump without modifying their preferences, and yet (c) the agent is not then representable as having modified their preferences in the relevant way?
Good point.
What I meant by updatelessness removes most of the justification is the reason given here at the very beginning of “Against Resolute Choice”. In order to make a money pump that leads the agent in a circle, the agent has to continue accepting trades around a full preference loop. But if it has decided on the entire plan beforehand, it will just do any plan that involves <1 trip around the preference loop. (Although it’s unclear how it would settle on such a plan, maybe just stopping its search after a given time). It won’t (I think?) choose any plan that does multiple loops, because they are strictly worse.
After choosing this plan though, I think it is representable as VNM rational, as you say. And I’m not sure what to do with this. It does seem important.
However, I think Scott’s argument here satisfies (a) (b) and (c). I think the independence axiom might be special in this respect, because the money pump for independence is exploiting an update on new information.
I don’t think agents that avoid the money pump for cyclicity are representable as satisfying VNM, at least holding fixed the objects of preference (as we should). Resolute choosers with cyclic preferences will reliably choose B over A- at node 3, but they’ll reliably choose A- over B if choosing between these options ex nihilo. That’s not VNM representable, because it requires that the utility of A- be greater than the utility of B and. that the utility of B be greater than the utility of A-
Perhaps I’m misusing the word “representable”? But what I meant was that any single sequence of actions generate by the agent could also have been generated by an outcome-utility maximizer (that has the same world model). This seems like the relevant definition, right?
That’s not right. As I say in another comment:
Or consider another example. The agent trades A for B, then B for A, then declines to trade A for B+. That’s compatible with the Caprice rule, but not with complete preferences.
Or consider the pattern of behaviour that (I elsewhere argue) can make agents with incomplete preferences shutdownable. Agents abiding by the Caprice rule can refuse to pay costs to shift probability mass between A and B, and refuse to pay costs to shift probability mass between A and B+. Agents with complete preferences can’t do that.
[I’m going to use the phrase ‘resolute choice’ rather than ‘updatelessness.’ That seems like a more informative and less misleading description of the relevant phenomenon: making a plan and sticking to it. You can stick to a plan even if you update your beliefs. Also, in the posts on UDT, ‘updatelessness’ seems to refer to something importantly distinct from just making a plan and sticking to it.]
That’s right, but the drawbacks of resolute choice depend on the money pump to which you apply it. As Gustafsson notes, if an agent uses resolute choice to avoid the money pump for cyclic preferences, that agent has to choose against their strict preferences at some point. For example, they have to choose B at node 3 in the money pump below, even though—were they facing that choice ex nihilo—they’d prefer to choose A-.
There’s no such drawback for agents with incomplete preferences using resolute choice. As I note in this post, agents with incomplete preferences using resolute choice need never choose against their strict preferences. The agent’s past plan only has to serve as a tiebreaker: forcing a particular choice between options between which they’d otherwise lack a preference. For example, they have to choose B at node 2 in the money pump below. Were they facing that choice ex nihilo, they’d lack a preference between B and A-.
Are you saying that my description (following) is incorrect?
Or are you saying that it is correct, but you disagree that this implies that it is “behaviorally indistinguishable from an agent with complete preferences”? If this is the case, then I think we might disagree on the definition of “behaviorally indistinguishable”? I’m using it like: If you observe a single sequence of actions from this agent (and knowing the agent’s world model), can you construct a utility function over outcomes that could have produced that sequence.
This is compatible with a resolute outcome-utility maximizer (for whom A is a maxima). There’s no rule that says an agent must take the shortest route to the same outcome (right?).
Sure, but why is that a drawback? It can’t be money pumped, right? Agents following resolute choice often choose against their local strict preferences in other decision problems. (E.g. Newcomb’s). And this is considered an argument in favour of resolute choice.