[Edit: I think I misinterpreted EJT in a way that invalidates some of this comment, see downthread comment clarifying this].
That is really helpful, thanks. I had been making a mistake, in that I thought that there was an argument from just “the agent thinks it’s possible the agent will run into a money pump” that concluded “the agent should complete that preference in advance”. But I was thinking sloppily and accidentally sometimes equivocating between pref-gaps and indifference. So I don’t think this argument works by itself, but I think it might be made to work with an additional assumption.
One intuition that I find convincing is that if I found myself at outcome A in the single sweetening money pump, I would regret having not made it to A+. This intuition seems to hold even if I imagine A and B to be of incomparable value.
In order to avoid this regret, I would try to become the sort of agent that never found itself in that position. I can see that if I always follow the Caprice rule, then it’s a little weird to regret not getting A+, because that isn’t a counterfactually available option (counterfacting on decision 1). But this feels like I’m being cheated. I think the reason that if feels like I’m being cheated is that I feel like getting to A+ should be a counterfactually available option.
One way to make it a counterfactually available option in the thought experiment is to introduce another choice before choice 1 in the decision tree. The new choice (0), is the choice about whether to maintain the same decision algorithm (call this incomplete), or complete the preferential gap between A and B (call this complete).
I think the choice complete statewise dominates incomplete. This is because the choice incomplete results in a lottery {B: qp, A+: q(1−p), A:(1−q)} for q<1.[1] However, the choice complete results in the lottery {B: p, A+: (1−p), A:0}.
Do you disagree with this? I think this allows us to create a money pump, by charging the agent $ϵ for the option to complete its own preferences.
The statewise pseudodominance relation is cyclic, so the Statewise Pseudodominance Principle would lead to cyclic preferences.
This still seems wrong to me, because I see lotteries as being an object whose purpose is to summarize random variables and outcomes. So it’s weird to compare lotteries that depend on the same random variables (they are correlated), as if they are independent. This seems like a sidetrack though, and it’s plausible to me that I’m just confused about your definitions here.
Letting p be the probability that the agent chooses 2A+ and (1−p) the probability the agent chooses 2B (following your comment above). And q is defined similarly, for choice 1.
I made a mistake again. As described above, complete only pseudodominates incomplete.
But this is easily patched with the trick described in the OP. So we need the choice complete to make two changes to the downstream decisions. First, change decision 1 to always choose up (as before), second, change the distribution of Decision 2 to {1−q(1−p), q(1−p)}, because this keeps the probability of B constant. Fixed diagram:
Now the lottery for complete is {B: q(1−p), A+: 1−q(1−p), A:0}, and the lottery for incomplete is {B: q(1−p), A+: pq, A:(1−q)}. So overall, there is a pure shift of probability from A to A+. [Edit 23/7: hilariously, I still had the probabilities wrong, so fixed them, again].
I think the above money pump works, if the agent sometimes chooses the A path, but I was incorrect in thinking that the caprice rule sometimes chooses the A path.
I misinterpreted one of EJT’s comments as saying it might choose the A path. The last couple of days I’ve been reading through some of the sources he linked to in the original “there are no coherence theorems” post and one of them (Gustafsson) made me realize I was interpreting him incorrectly, by simplifying the decision tree in a way that doesn’t make sense. I only realized this yesterday.
Now I think that the caprice rule is essentially equivalent to updatelessness. If I understand correctly, it would be equivalent to 1. choosing the best policy by ranking them in the partial order of outcomes (randomizing over multiple maxima), then 2. implementing that policy without further consideration. And this makes it immune to money pumps and renders any self-modification pointless. It also makes it behaviorally indistinguishable from an agent with complete preferences, as far as I can tell. The same updatelessness trick seems to apply to all money pump arguments. It’s what scott uses in this post to avoid the independence money pump.
So currently I’m thinking updatelessness removes most of the justification for the VNM axioms (including transitivity!). But I’m confused because updateless policies still must satisfy local properties like “doesn’t waste resources unless it helps achieve the goal”, which is intuitively what the money pump arguments represent. So there must be some way to recover properties like this. Maybe via John’s approach here. But I’m only maybe 80% sure of my new understanding, I’m still trying to work through it all.
[Edit: I think I misinterpreted EJT in a way that invalidates some of this comment, see downthread comment clarifying this].
That is really helpful, thanks. I had been making a mistake, in that I thought that there was an argument from just “the agent thinks it’s possible the agent will run into a money pump” that concluded “the agent should complete that preference in advance”. But I was thinking sloppily and accidentally sometimes equivocating between pref-gaps and indifference. So I don’t think this argument works by itself, but I think it might be made to work with an additional assumption.
One intuition that I find convincing is that if I found myself at outcome A in the single sweetening money pump, I would regret having not made it to A+. This intuition seems to hold even if I imagine A and B to be of incomparable value.
In order to avoid this regret, I would try to become the sort of agent that never found itself in that position. I can see that if I always follow the Caprice rule, then it’s a little weird to regret not getting A+, because that isn’t a counterfactually available option (counterfacting on decision 1). But this feels like I’m being cheated. I think the reason that if feels like I’m being cheated is that I feel like getting to A+ should be a counterfactually available option.
One way to make it a counterfactually available option in the thought experiment is to introduce another choice before choice 1 in the decision tree. The new choice (0), is the choice about whether to maintain the same decision algorithm (call this incomplete), or complete the preferential gap between A and B (call this complete).
I think the choice complete statewise dominates incomplete. This is because the choice incomplete results in a lottery {B: qp, A+: q(1−p), A:(1−q)} for q<1.[1] However, the choice complete results in the lottery {B: p, A+: (1−p), A:0}.
Do you disagree with this? I think this allows us to create a money pump, by charging the agent $ϵ for the option to complete its own preferences.
This still seems wrong to me, because I see lotteries as being an object whose purpose is to summarize random variables and outcomes. So it’s weird to compare lotteries that depend on the same random variables (they are correlated), as if they are independent. This seems like a sidetrack though, and it’s plausible to me that I’m just confused about your definitions here.
Letting p be the probability that the agent chooses 2A+ and (1−p) the probability the agent chooses 2B (following your comment above). And q is defined similarly, for choice 1.
I made a mistake again. As described above, complete only pseudodominates incomplete.
But this is easily patched with the trick described in the OP. So we need the choice complete to make two changes to the downstream decisions. First, change decision 1 to always choose up (as before), second, change the distribution of Decision 2 to {1−q(1−p), q(1−p)}, because this keeps the probability of B constant. Fixed diagram:
Now the lottery for complete is {B: q(1−p), A+: 1−q(1−p), A:0}, and the lottery for incomplete is {B: q(1−p), A+: pq, A:(1−q)}. So overall, there is a pure shift of probability from A to A+.
[Edit 23/7: hilariously, I still had the probabilities wrong, so fixed them, again].
I think the above money pump works, if the agent sometimes chooses the A path, but I was incorrect in thinking that the caprice rule sometimes chooses the A path.
I misinterpreted one of EJT’s comments as saying it might choose the A path. The last couple of days I’ve been reading through some of the sources he linked to in the original “there are no coherence theorems” post and one of them (Gustafsson) made me realize I was interpreting him incorrectly, by simplifying the decision tree in a way that doesn’t make sense. I only realized this yesterday.
Now I think that the caprice rule is essentially equivalent to updatelessness. If I understand correctly, it would be equivalent to 1. choosing the best policy by ranking them in the partial order of outcomes (randomizing over multiple maxima), then 2. implementing that policy without further consideration. And this makes it immune to money pumps and renders any self-modification pointless. It also makes it behaviorally indistinguishable from an agent with complete preferences, as far as I can tell.
The same updatelessness trick seems to apply to all money pump arguments. It’s what scott uses in this post to avoid the independence money pump.
So currently I’m thinking updatelessness removes most of the justification for the VNM axioms (including transitivity!). But I’m confused because updateless policies still must satisfy local properties like “doesn’t waste resources unless it helps achieve the goal”, which is intuitively what the money pump arguments represent. So there must be some way to recover properties like this. Maybe via John’s approach here.
But I’m only maybe 80% sure of my new understanding, I’m still trying to work through it all.