Wait, I can construct a money pump for that situation. First let the agent choose between A and C. If there’s a preferential gap, the agent should sometimes choose C. Then let the agent pay a penny to upgrade from C to B. Then let the agent pay a penny to upgrade from B to A. The agent is now where it could have been to begin with by choosing A in the first place, but 2 cents poorer.
Even if we ditch the completeness axiom, it sure seems like money pump arguments require us to assume a partial order.
So this won’t work if the agent knows in advance what trades they’ll be offered and is capable of reasoning by backward induction. In that case, the agent will reason that they’d choose A-2p over B-1p if they reached that node, and would choose B-1p over C if they reached that node. So (they will reason), the choice between A and C is actually a choice between A and A-2p, and so they will reliably choose A.
And plausibly we should make assumptions like ‘the agent knows in advance what trades they will be offered’ and ‘the agent is capable of backward induction’ if we’re arguing about whether agents are rationally required to conform their preferences to the VNM axioms.
(If the agent doesn’t know in advance what trades they will be offered or is incapable of backward induction, then their pursuit of a dominated strategy need not indicate any defect in their preferences. Their pursuit of a dominated strategy can instead be blamed on their lack of knowledge and/or reasoning ability.)
That said, I’ve recently become less convinced that ‘knowing trades in advance’ is a reasonable assumption in the context of predicting the behaviour of advanced artificial agents. And your money-pump seems to work if we assume that the agent doesn’t know what trades they will be offered in advance. So maybe we do in fact have reason to expect that advanced artificial agents will have transitive preferences. (I say ‘maybe’ because there are some other relevant considerations pushing the other way, discussed in a paper-in-progress by Adam Bales.)
I don’t know, this still seems kind of sketchy to me. Say we change the experiment so that it costs the agent a penny to choose A in the initial choice: it will still take that choice, since A-1p is still preferable to A-2p. Compare this to a game where the agent can freely choose between A and C, and there’s no cost in pennies to either choice. Since there’s a preferential gap between A and C, the agent will sometimes pick A and sometimes pick C. In the first game, on the other hand the agent always picks A. Yet in the first game, not only is picking A more costly, but we’ve only added options for the agent if it picks C. In other words, an agent that has A>B, B>C, and A~C sure looks like it’s paying to take options away from itself, since adding options makes it less likely to pick C, even when it costs a penny to avoid it.
Nice! This is a cool case. The behaviour does indeed seem weird. I’m inclined to call it irrational. But the agent isn’t pursuing a dominated strategy: in neither game does the agent settle on an option that they strictly disprefer to some other available option.
This discussion is interesting and I’m happy to keep having it, but perhaps it’s worth saying (if not for your sake then for other readers) that this is a side-thread. The main point of the post is that there are no money-pumps for Completeness. I think that there are probably no money-pumps for Transitivity either, but it’s the claim about Completeness that I really want to defend.
Cool. For me personally, I think that paying to avoid being given more options looks enough like being dominated that I’d want to keep the axiom of transitivity around, even if it’s not technically a money pump.
So in the case where we have transitivity but no completeness, it seems kind of like there might be a weaker coherence theorem, where the agent’s behaviour can be described by rolling a dice to pick a utility function before beginning a game, and then subsequently playing according to that utility function. Under this interpretation, if A > B then that means that A is preferred to B under all utility functions the agent could pick, while a preferential gap between A and B means that sometimes A will be ranked higher and sometimes B will be ranked higher, depending on which utility function the die roll happens to land on.
Does this match your intuition? Is there an obvious counterexample to this “coherence conjecture”?
This is cool. I don’t think violations of continuity are also in general exploitable, but I’d guess you should also be able to replace continuity with something weaker from Russell and Isaacs, 2020, just enough to rule out St. Petersburg-like lotteries, specifically any one of Countable Independence (which can also replace independence), the Extended Outcome Principle (which can also replace independence) or Limitedness, and then replace the real-valued utility functions with utility functions representable by “lexicographically ordered ordinal sequences of bounded real utilities”.
Wait, I can construct a money pump for that situation. First let the agent choose between A and C. If there’s a preferential gap, the agent should sometimes choose C. Then let the agent pay a penny to upgrade from C to B. Then let the agent pay a penny to upgrade from B to A. The agent is now where it could have been to begin with by choosing A in the first place, but 2 cents poorer.
Even if we ditch the completeness axiom, it sure seems like money pump arguments require us to assume a partial order.
What am I missing?
So this won’t work if the agent knows in advance what trades they’ll be offered and is capable of reasoning by backward induction. In that case, the agent will reason that they’d choose A-2p over B-1p if they reached that node, and would choose B-1p over C if they reached that node. So (they will reason), the choice between A and C is actually a choice between A and A-2p, and so they will reliably choose A.
And plausibly we should make assumptions like ‘the agent knows in advance what trades they will be offered’ and ‘the agent is capable of backward induction’ if we’re arguing about whether agents are rationally required to conform their preferences to the VNM axioms.
(If the agent doesn’t know in advance what trades they will be offered or is incapable of backward induction, then their pursuit of a dominated strategy need not indicate any defect in their preferences. Their pursuit of a dominated strategy can instead be blamed on their lack of knowledge and/or reasoning ability.)
That said, I’ve recently become less convinced that ‘knowing trades in advance’ is a reasonable assumption in the context of predicting the behaviour of advanced artificial agents. And your money-pump seems to work if we assume that the agent doesn’t know what trades they will be offered in advance. So maybe we do in fact have reason to expect that advanced artificial agents will have transitive preferences. (I say ‘maybe’ because there are some other relevant considerations pushing the other way, discussed in a paper-in-progress by Adam Bales.)
I don’t know, this still seems kind of sketchy to me. Say we change the experiment so that it costs the agent a penny to choose A in the initial choice: it will still take that choice, since A-1p is still preferable to A-2p. Compare this to a game where the agent can freely choose between A and C, and there’s no cost in pennies to either choice. Since there’s a preferential gap between A and C, the agent will sometimes pick A and sometimes pick C. In the first game, on the other hand the agent always picks A. Yet in the first game, not only is picking A more costly, but we’ve only added options for the agent if it picks C. In other words, an agent that has A>B, B>C, and A~C sure looks like it’s paying to take options away from itself, since adding options makes it less likely to pick C, even when it costs a penny to avoid it.
Nice! This is a cool case. The behaviour does indeed seem weird. I’m inclined to call it irrational. But the agent isn’t pursuing a dominated strategy: in neither game does the agent settle on an option that they strictly disprefer to some other available option.
This discussion is interesting and I’m happy to keep having it, but perhaps it’s worth saying (if not for your sake then for other readers) that this is a side-thread. The main point of the post is that there are no money-pumps for Completeness. I think that there are probably no money-pumps for Transitivity either, but it’s the claim about Completeness that I really want to defend.
Cool. For me personally, I think that paying to avoid being given more options looks enough like being dominated that I’d want to keep the axiom of transitivity around, even if it’s not technically a money pump.
So in the case where we have transitivity but no completeness, it seems kind of like there might be a weaker coherence theorem, where the agent’s behaviour can be described by rolling a dice to pick a utility function before beginning a game, and then subsequently playing according to that utility function. Under this interpretation, if A > B then that means that A is preferred to B under all utility functions the agent could pick, while a preferential gap between A and B means that sometimes A will be ranked higher and sometimes B will be ranked higher, depending on which utility function the die roll happens to land on.
Does this match your intuition? Is there an obvious counterexample to this “coherence conjecture”?
Your coherence conjecture sounds good! It sounds like it roughly matches this theorem:
Screenshot is from this paper.
This is cool. I don’t think violations of continuity are also in general exploitable, but I’d guess you should also be able to replace continuity with something weaker from Russell and Isaacs, 2020, just enough to rule out St. Petersburg-like lotteries, specifically any one of Countable Independence (which can also replace independence), the Extended Outcome Principle (which can also replace independence) or Limitedness, and then replace the real-valued utility functions with utility functions representable by “lexicographically ordered ordinal sequences of bounded real utilities”.
This also looks like a generalization of stochastic dominance.
“paying to avoid being given more options looks enough like being dominated that I’d want to keep the axiom of transitivity around”
Maybe offtopic but paying to avoid being given more options is a common strategy in negotiation.
It’s not a money pump, because money pump implies infinite cycle of profit. If your loses are bounded, you are fine.