P.S. Here’s the aforementioned “Aumann (1962)” (yes, that very same Robert J. Aumann)—a paper called “Utility Theory without the Completeness Axiom”. Aumann writes in plain language wherever possible, and the paper is very readable. It includes this line:
Of all the axioms of utility theory, the completeness axiom is perhaps the
most questionable.[8] Like others of the axioms, it is inaccurate as a description
of real life; but unlike them, we find it hard to accept even from the
normative viewpoint.
The full elaboration for this (perhaps quite shocking) comment is too long to quote; I encourage anyone who’s at all interested in utility theory to read the paper.
I happened upon this old thread, and found the discussion intriguing. Thanks for posting these references! Unless I’m mistaken, it sounds like you’ve discussed this topic a lot on LW but have never made a big post detailing your whole perspective. Maybe that would be useful! At least I personally find discussions of applicability/generalizability of VNM and other rationality axioms quite interesting.
Indeed, I think I recently ran into another old comment of yours in which you made a remark about how Dutch Books only hold for repeated games? I don’t recall the details now.
I have some comments on the preceding discussion. You said:
It would be rather audacious to claim that this is true for each of the four axioms. For instance, do please demonstrate how you would Dutch-book an agent that does not conform to the completeness axiom!
For me, it seems that transitivity and completeness are on an equally justified footing, based on the classic money-pump argument.
Just to keep things clear, here is how I think about the details. There are outcomes. Then there are gambles, which we will define recursively. An outcome counts as a gamble for the sake of the base case of our recursion. For gambles A and B, pA+(1-p)B also counts as a gamble, where p is a real number in the range [0,1].
Now we have a preference relation > on our gambles. I understand its negation to be ≤; saying ¬(A>B) is the same thing as A≤B. The indifference relation, A∼B is just the same thing as (A≤B)&(B≤A).
This is different than the development on wikipedia, where ~ is defined separately. But I think it makes more sense to define > and then define ~ from that. A>B can be understood as “definitely choose A when given the choice between A and B”. ~ then represents indifference as well as uncertainty like the kind you describe when you discuss bounded rationality.
From this starting point, it’s clear that either A<B, or B<A, or A~B. This is just a way of saying “either A<B or B<A or neither”. What’s important about the completeness axiom is the assumption that exactly one of these hold; this tells us that we cannot have both A<B and B<A.
But this is practically the same as circular preferences A<B<C<A, which transitivity outlaws. It’s just a circle of length 2.
The classic money-pump against circularity is that if we have circular preferences, someone can charge us for making a round trip around the circle, swapping A for B for C for A again. They leave us in the same position we started, less some money. They can then do this again and again, “pumping” all the money out of us.
Personally I find the this argument extremely metaphysically weird, for several reasons.
The money-pumper must be God, to be able to swap arbitrary A for B, and B for C, etc.
But furthermore, the agent must not understand the true nature of the money-pumper. When God asks about swapping A for B, the agent thinks it’ll get B in the end, and makes the decision accordingly. Yet, God proceeds to then ask a new question, offering to swap B for C. So God doesn’t actually put the agent in universe B; rather, God puts the agent in “B+God”, a universe with the possibility of B, but also a new offer from God, namely, to move on to C. So God is actually fooling the agent, making an offer of B but really giving the agent something different than B. Bad decision-making should not count against the agent if the agent was mislead in such a manner!
It’s also pretty weird that we can end up “in the same situation, but with less money”. If the outcomes A,B,C were capturing everything about the situation, they’d include how much money we had!
I have similar (but less severe) objections to Dutch-book arguments.
However, I also find the argument extremely practically applicable, so much so that I can excuse the metaphysical weirdness. I have come to think of Dutch-book and money-pump arguments as illustrative of important types of (in)consistency rather than literal arguments.
OK, why do I find money-pumps practical?
Simply put, if I have a loop in my preferences, then I will waste a lot of time deliberating. The real money-pump isn’t someone taking advantage of me, but rather, time itself passing.
What I find is that I get stuck deliberating until I can find a way to get rid of the loop. Or, if I “just choose randomly”, I’m stuck with a yucky dissatisfied feeling (I have regret, because I see another option as better than the one I chose).
This is equally true of three-choice loops and two-choice loops. So, transitivity and completeness seem equally well-justified to me.
Stuart Armstrong argues that there is a weak money pump for the independence axiom. I made a very technical post (not all of which seems to render correctly on LessWrong :/) justifying as much as I could with money-pump/dutch-book arguments, and similarly got everything except continuity.
I regard continuity as not very theoretically important, but highly applicable in practice. IE, I think the pure theory of rationality should exclude continuity, but a realistic agent will usually have continuous values. The reason for this is again because of deliberation time.
If we drop continuity, we get a version of utility theory with infinite and infinitesimal values. This is perfectly fine, has the advantage of being more general, and is in some sense more elegant. To reference the OP, continuity is definitely just boilerplate; we get a nice generalization if we want to drop it.
However, a real agent will ignore its own infinitesimal preferences, because it’s not worth spending time thinking about that. Indeed, it will almost always just think about the largest infinity in its preferences. This is especially true if we assume that the agent places positive probability on a really broad class of things, which again seems true of capable agents in practice. (IE, if you have infinities in your values, and a broad probability distribution, you’ll be Pascal-mugged—you’ll only think of the infinite payoffs, neglecting finite payoffs.)
So all of the axioms except independence have what appear to me to be rather practical justifications, and independence has a weak money-pump justification (which may or may not translate to anything practical).
Correction: I now see that my formulation turns the question of completeness into a question of transitivity of indifference. An “incomplete” preference relation should not be understood as one in which allows strict preferences to go in both directions (which is what I interpret them as, above) but rather, a preference relation in which the ≤ relation (and hence the ∼ relation) is not transitive.
In this case, we can distinguish between ~ and “gaps”, IE, incomparable A and B. ~ might be transitive, but this doesn’t bridge across the gaps. So we might have a preference chain A>B>C and a chain X>Y>Z, but not have any way to compare between the two chains.
In my formulation, which lumps together indifference and gaps, we can’t have this two-chain situation. If A~X, then we must have A>Y, since X>Y, by transitivity of ≥.
So what would be a completeness violation in the wikipedia formulation becomes a transitivity violation in mine.
But notice that I never argued for the transitivity of ~ or ≥ in my comment; I only argued for the transitivity of >.
I don’t think a money-pump argument can be offered for transitivity here.
However, I took a look at the paper by Aumann which you cited, and I’m fairly happy with the generalization of VNM therein! Dropping uniqueness does not seem like a big cost. This seems like more of an example of John Wentworth’s “boilerplate” point, rather than a counterexample.
P.S. Here’s the aforementioned “Aumann (1962)” (yes, that very same Robert J. Aumann)—a paper called “Utility Theory without the Completeness Axiom”. Aumann writes in plain language wherever possible, and the paper is very readable. It includes this line:
The full elaboration for this (perhaps quite shocking) comment is too long to quote; I encourage anyone who’s at all interested in utility theory to read the paper.
I happened upon this old thread, and found the discussion intriguing. Thanks for posting these references! Unless I’m mistaken, it sounds like you’ve discussed this topic a lot on LW but have never made a big post detailing your whole perspective. Maybe that would be useful! At least I personally find discussions of applicability/generalizability of VNM and other rationality axioms quite interesting.
Indeed, I think I recently ran into another old comment of yours in which you made a remark about how Dutch Books only hold for repeated games? I don’t recall the details now.
I have some comments on the preceding discussion. You said:
For me, it seems that transitivity and completeness are on an equally justified footing, based on the classic money-pump argument.
Just to keep things clear, here is how I think about the details. There are outcomes. Then there are gambles, which we will define recursively. An outcome counts as a gamble for the sake of the base case of our recursion. For gambles A and B, pA+(1-p)B also counts as a gamble, where p is a real number in the range [0,1].
Now we have a preference relation > on our gambles. I understand its negation to be ≤; saying ¬(A>B) is the same thing as A≤B. The indifference relation, A∼B is just the same thing as (A≤B)&(B≤A).
This is different than the development on wikipedia, where ~ is defined separately. But I think it makes more sense to define > and then define ~ from that. A>B can be understood as “definitely choose A when given the choice between A and B”. ~ then represents indifference as well as uncertainty like the kind you describe when you discuss bounded rationality.
From this starting point, it’s clear that either A<B, or B<A, or A~B. This is just a way of saying “either A<B or B<A or neither”. What’s important about the completeness axiom is the assumption that exactly one of these hold; this tells us that we cannot have both A<B and B<A.
But this is practically the same as circular preferences A<B<C<A, which transitivity outlaws. It’s just a circle of length 2.
The classic money-pump against circularity is that if we have circular preferences, someone can charge us for making a round trip around the circle, swapping A for B for C for A again. They leave us in the same position we started, less some money. They can then do this again and again, “pumping” all the money out of us.
Personally I find the this argument extremely metaphysically weird, for several reasons.
The money-pumper must be God, to be able to swap arbitrary A for B, and B for C, etc.
But furthermore, the agent must not understand the true nature of the money-pumper. When God asks about swapping A for B, the agent thinks it’ll get B in the end, and makes the decision accordingly. Yet, God proceeds to then ask a new question, offering to swap B for C. So God doesn’t actually put the agent in universe B; rather, God puts the agent in “B+God”, a universe with the possibility of B, but also a new offer from God, namely, to move on to C. So God is actually fooling the agent, making an offer of B but really giving the agent something different than B. Bad decision-making should not count against the agent if the agent was mislead in such a manner!
It’s also pretty weird that we can end up “in the same situation, but with less money”. If the outcomes A,B,C were capturing everything about the situation, they’d include how much money we had!
I have similar (but less severe) objections to Dutch-book arguments.
However, I also find the argument extremely practically applicable, so much so that I can excuse the metaphysical weirdness. I have come to think of Dutch-book and money-pump arguments as illustrative of important types of (in)consistency rather than literal arguments.
OK, why do I find money-pumps practical?
Simply put, if I have a loop in my preferences, then I will waste a lot of time deliberating. The real money-pump isn’t someone taking advantage of me, but rather, time itself passing.
What I find is that I get stuck deliberating until I can find a way to get rid of the loop. Or, if I “just choose randomly”, I’m stuck with a yucky dissatisfied feeling (I have regret, because I see another option as better than the one I chose).
This is equally true of three-choice loops and two-choice loops. So, transitivity and completeness seem equally well-justified to me.
Stuart Armstrong argues that there is a weak money pump for the independence axiom. I made a very technical post (not all of which seems to render correctly on LessWrong :/) justifying as much as I could with money-pump/dutch-book arguments, and similarly got everything except continuity.
I regard continuity as not very theoretically important, but highly applicable in practice. IE, I think the pure theory of rationality should exclude continuity, but a realistic agent will usually have continuous values. The reason for this is again because of deliberation time.
If we drop continuity, we get a version of utility theory with infinite and infinitesimal values. This is perfectly fine, has the advantage of being more general, and is in some sense more elegant. To reference the OP, continuity is definitely just boilerplate; we get a nice generalization if we want to drop it.
However, a real agent will ignore its own infinitesimal preferences, because it’s not worth spending time thinking about that. Indeed, it will almost always just think about the largest infinity in its preferences. This is especially true if we assume that the agent places positive probability on a really broad class of things, which again seems true of capable agents in practice. (IE, if you have infinities in your values, and a broad probability distribution, you’ll be Pascal-mugged—you’ll only think of the infinite payoffs, neglecting finite payoffs.)
So all of the axioms except independence have what appear to me to be rather practical justifications, and independence has a weak money-pump justification (which may or may not translate to anything practical).
Correction: I now see that my formulation turns the question of completeness into a question of transitivity of indifference. An “incomplete” preference relation should not be understood as one in which allows strict preferences to go in both directions (which is what I interpret them as, above) but rather, a preference relation in which the ≤ relation (and hence the ∼ relation) is not transitive.
In this case, we can distinguish between ~ and “gaps”, IE, incomparable A and B. ~ might be transitive, but this doesn’t bridge across the gaps. So we might have a preference chain A>B>C and a chain X>Y>Z, but not have any way to compare between the two chains.
In my formulation, which lumps together indifference and gaps, we can’t have this two-chain situation. If A~X, then we must have A>Y, since X>Y, by transitivity of ≥.
So what would be a completeness violation in the wikipedia formulation becomes a transitivity violation in mine.
But notice that I never argued for the transitivity of ~ or ≥ in my comment; I only argued for the transitivity of >.
I don’t think a money-pump argument can be offered for transitivity here.
However, I took a look at the paper by Aumann which you cited, and I’m fairly happy with the generalization of VNM therein! Dropping uniqueness does not seem like a big cost. This seems like more of an example of John Wentworth’s “boilerplate” point, rather than a counterexample.