The biggest problematic unstated assumption behind applying VNM-rationality to humans, I think, is the assumption that we’re actually trying to maximize something.
To elaborate, the VNM theorem defines preferences by the axiom of completeness, which states that for any two lotteries A and B, one of the following holds: A is preferred to B, B is preferred to A, or one is indifferent between them.
So basically, a “preference” as defined by the axioms is a function that (given the state of the agent and the state of the world in general) outputs an agent’s decision between two or more choices. Now suppose that the agent’s preferences violate the Von Neumann-Morgenstern axioms, so that in one situation it prefers to make a deal that causes it to end up with an apple rather than an orange, and in another situation it prefers to make a deal that causes it to end up with an orange rather than an apple. Is that an argument against having circular preferences?
By itself, it’s not. It simply establishes that the function that outputs the agent’s actions behaves differently in different situations. Now the normal way to establish that this is bad is to assume that all choices are between monetary payouts, and that an agent with inconsistent preferences can be Dutch Booked and made to lose money. An alternative way, which doesn’t require us to assume that all the choices are between monetary payouts, is to construct a series of trades between resources that leaves us with less resources than when we started.
Stated that way, this sounds kinda bad. But then there are things that kind of fit that description, but which we would intuitively think of as good. For instance, some time back I asked:
Suppose someone has a preference to have sex each evening, and is in a relationship with someone what a similar level of sexual desire. So each evening they get into bed, undress, make love, get dressed again, get out of bed. Repeat the next evening.
How is this different from having exploitable circular preferences? After all, the people involved clearly have cycles in their preferences—first they prefer getting undressed to not having sex, after which they prefer getting dressed to having (more) sex. And they’re “clearly” being the victims of a Dutch Book, too—they keep repeating this set of trades every evening, and losing lots of time because of that.
In response, I was told that
The circular preferences that go against the axioms of utility theory, and which are Dutch book exploitable, are not of the kind “I prefer A to B at time t1 and B to A at time t2”, like the ones of your example. They are more like “I prefer A to B and B to C and C to A, all at the same time”.
The couple, if they had to pay a third party a cent to get undressed and then a cent to get dressed, would probably do it and consider it worth it—they end up two cents short but having had an enjoyable experience. Nothing irrational about that. To someone with the other “bad” kind of circular preferences, we can offer a sequence of trades (first A for B and a cent, then C for A and a cent, then B for C and a cent) after which they end up three cents short but otherwise exactly as they started (they didn’t actually obtain enjoyable experiences, they made all the trades before anything happened). It is difficult to consider this rational.
But then I asked that, if we accept this, then what real-life situation does count as an actual circular preference in the VNM sense, given that just about every potential circularity that I can think of is the kind “I prefer A to B at time t1 and B to A at time t2”? And I didn’t get very satisfactory replies.
Intuitively, there are a lot of real-life situations that feel kind of like losing out due to inconsistent preferences, like someone who wants to get into a relationship when he’s single and then wants to be single when he gets into a relationship, but there our actual problem is that the person spends a lot of time being unhappy, rather than with the fact that he makes different choices in different situations. Whereas with the couple, we think that’s fine because they get enjoyment from the “trades”.
The general problem that I’m trying to get at is that in order to hold up VNM rationality as a normative standard, we would need to have a meta-preference: a preference over preferences, stating that it would be better to have preferences that lead to some particular outcomes. The standard Dutch Book example kind of smuggles in that assumption by the way that it talks about money, and thus makes us think that we are in a situation where we are only trying to maximize money and care about nothing else. And if you really are trying to only maximize a single concrete variable or resource and care about nothing else, then you really should try to make sure that your choices follow the VNM axioms. If you run a betting office, then do make sure that nobody can Dutch Book you.
But we don’t have such a clear normative standard for life in general. It would be reasonable to try to construct an argument for why the couple having sex were rational but the person who kept vacillating about being in a relationship was irrational by suggesting that the couple got happiness whereas the other person was unhappy… but we also care about other things than justhappiness (or pleasure) and thus aren’t optimizing just for pleasure either. And unless you’re a hedonistic utilitarian, you’re unlikely to say that we should optimize only for pleasure either.
So basically, if you want to say that people should be VNM-rational, then you need to have some specific set of values or goals that you think people should strive towards. If you don’t have that, then VNM-rationality is basically irrelevant aside for the small set of special cases where people really do have a clear explicit goal that’s valued above other things.
Now suppose that the agent’s preferences violate the Von Neumann-Morgenstern axioms, so that in one situation it prefers to make a deal that causes it to end up with an apple rather than an orange, and in another situation it prefers to make a deal that causes it to end up with an orange rather than an apple. Is that an argument against having circular preferences?
I’m not sure I follow in what sense this is a violation of the vNM axioms. A vNM agent has preferences over world-histories; in general one can’t isolate the effect of having an apple vs. having an orange without looking at how that affects the entire future history of the world.
Right, I was trying to say “it prefers an apple to an orange and an orange to an apple in such a way that does violate the axioms”. But I was unsure of what example to actually give of that, since I’m unsure of what real-life situations really would violate the axioms.
The example that comes to mind to show the how the sex thing isn’t a problem is that of a robot car with a goal to drive as many miles as possible. Every day it will burn through all its fuel and fuel up. Right after it fuels up, it will have no desire for further fuel—more fuel simply does not help it go further at this point, and forcing it can be detrimental. Clearly not contradictory
You could have a similar situation with a couple wanting sex iff they haven’t had sex in a day, or wanting an orange if you’ve just eaten an apple but wanting an apple if you’ve just eaten an orange.
To strictly show that something violates vNM axioms, you’d have to show that this behavior (in context) can’t be fulfilling any preferences better than other options that the agent is aware of—or at least be able to argue that the revealed utility function is contrived and unlikely to hold up in other situations (not what the agent “really wants”).
Constantly wanting what one doesn’t have can have this defect. If I keep paying you to switch my apple for your orange and back (without actually eating either), then you have a decent case, if you’re pretty confident I’m not actually fulfilling my desire to troll you ;)
The “want’s a relationship when single” and “wants to be single when not” thing does look like such a violation to me. If you let him flip flop as often as he desires, he’s not going to end up happily endorsing his past actions. If you offered him a pill that would prevent him from flip flopping, he very well may take it. So there’s a contradiction there.
To bring human-specific psychology into it, its not that his inherent desires are contradictory, but that he wants something like “freedom”, which he doesn’t know how to get in a relationship and something like “intimacy”, which he doesn’t know how to get while single. It’s not that he want’s intimacy when single and freedom when not, it’s that he wants both always, but the unfulfilled need is the salient one.
Picture me standing on your left foot. “Oww! Get off my left foot!”. Then I switch to the right “Ahh! Get off my right foot!”. If you’re not very quick and/or the pain is overwhelming, it might take you a few iterations to realize the situation you’re in and to put the pain aside while you think of a way to get me off both feet (intimacy when single/freedom in a relationship). Or if you can’t have that, it’s another challenge to figure out what you want to do about it.
I wouldn’t model you as “just VNM-irrational”, even if your external behaviors are ineffective for everything you might want. I’d model you as “not knowing how to be VNM-rational in presence of strong pain(s)”, and would expect you to start behaving more effectively when shown how.
(and that is what I find, although showing someone how to be more rational is not trivial and “here’s a proof of the inconsistency of your actions now pick a side and stop feeling the desire for the other side” is almost never sufficient. You have to be able to model the specific way that they’re stuck and meet them there)
tl;dr: We’re not VNM-rational because we don’t know how to be, not because it’s not something we’re trying to do.
How do you distinguish his preferences being irrationally inconsistent (he is worse off from entering and leaving relationships repeatedly) from him truly wanting to be in relationships periodically (like how it’s rational to alternate between sleeping and waking rather than always doing one or the other)?
If there’s a pill that can make him stop switching (but doesn’t change his preferences), one of two things will happen: either he’ll never be in a relationship (prevented from entering), or he’ll stay in his current relationship forever (prevented from leaving). I wouldn’t be surprised if he dislikes both of the outcomes and decides not to take the pill.
The pill could instead change his preferences so that he no longer wants to flip-flop, but this argument seems too general—why not just give him a pill that makes him like everything much more than he does now? If my behavior is irrational, I should be able to make myself better off simply by changing my behavior, without having to modify my preferences.
How do you distinguish his preferences being irrationally inconsistent [...] from him truly wanting to be in relationships periodically[...]?
By talking to him. If it’s the latter, he’ll be able to say he prefers flip flopping like it’s just a matter of fact and if you probe into why he likes flip flopping, he’ll either have an answer that makes sense or he’ll talk about it in a way that shows that he is comfortable with not knowing. If it’s the former, he’ll probably say that he doesn’t like flip flopping, and if he doesn’t, it’ll leak signs of bullshit. It’ll come off like he’s trying to convince you of something because he is. And if you probe his answers for inconsistencies he’ll get hostile because he doesn’t want you to.
I’m not sure where you’re going with the “magic pill” hypotheticals, but I agree. The only thing I can think to add is that a lot of times the “winning behaviors” are largely mental and aren’t really available until you understand the situation better.
For example, if you break your foot and can’t get it x-rayed for a day, the right answer might be to just get some writing done—but if you try to force that behavior while you’re suffering, it’s not gonna go well. You have to actually be able to dismiss the pain signal before you have a mental space to write in.
I’m not sure where you’re going with the “magic pill” hypotheticals, but I agree.
I meant that if someone is behaving irrationally, forcing them to stop that behavior should make them better off. But it seems unlikely to me that forcing him to stay in his current relationship forever, or preventing him from ever entering a relationship (these are the two ways he can be stopped from flip-flopping) actually benefit him.
Forcing anyone to stay in their current relationship forever or forever preventing them from entering a relationship would be quite bad. In order to help him, he’d have to be doing worse than that.
The way to help him would be a bit trickier than that: let him have “good” relationships but not bad. Let him leave “bad” relationships but not good. And then control his mental behaviors so that he’s not allowed to spend time being miserable about his lack of options… (it’s hard to force rationality)
Controlling his mental behaviors would either be changing his preferences or giving him another option. For judging whether he is behaving irrationally, shouldn’t his preferences and set of choices be held fixed?
Relevant question: what does the cognitive science literature on choice-making, preference, and valuation have to say about all this? What mathematical structure actually does model human preferences?
Given that we run on top of neural networks and seem to use some Bayesian algorithms for certain forms of learning (citations available), I currently expect that our choice-making mechanisms might involve conditioning on features or states of our environment at some fundamental level.
My first guess would be that evolution has selected us for circular preferences that our genes money-pump so that we will propagate them. You can’t get off this ride while you’re human.
:-) I mean that if you embody human value, you’ll probably be a money-pumpable entity. Very few humans actually achieve an end to desire while still alive and mentally active.
I’ll take the challenge, then. I was already walking around thinking that the Four Noble Truths of the Buddha are a bunch of depressing bullshit that need to be fixed.
I’ve seen a bunch of different theories backed with varying amounts of experimental data—for instance, this, this and this—but I haven’t looked at them enough to tell which ones seem most correct.
That said, I still don’t remember running into any thorough discussion of what human preferences are, other than just “something that makes us make some choice in some situations”. I mention here that
some of our preferences are implicit in our automatic habits (the things that we show we value with our daily routines), some in the preprocessing of sensory data that our brains carry out (the things and ideas that are ”painted with” positive associations or feelings), and some in the configuration of our executive processes (the actions we actually end up doing in response to novel or conflicting situations).
And I’m a little skeptical of any theory of human preferences that doesn’t attempt to make any such breakdown and only takes a “black box” approach of looking at the outputs of our choice mechanism.
The biggest problematic unstated assumption behind applying VNM-rationality to humans, I think, is the assumption that we’re actually trying to maximize something.
To elaborate, the VNM theorem defines preferences by the axiom of completeness, which states that for any two lotteries A and B, one of the following holds: A is preferred to B, B is preferred to A, or one is indifferent between them.
So basically, a “preference” as defined by the axioms is a function that (given the state of the agent and the state of the world in general) outputs an agent’s decision between two or more choices. Now suppose that the agent’s preferences violate the Von Neumann-Morgenstern axioms, so that in one situation it prefers to make a deal that causes it to end up with an apple rather than an orange, and in another situation it prefers to make a deal that causes it to end up with an orange rather than an apple. Is that an argument against having circular preferences?
By itself, it’s not. It simply establishes that the function that outputs the agent’s actions behaves differently in different situations. Now the normal way to establish that this is bad is to assume that all choices are between monetary payouts, and that an agent with inconsistent preferences can be Dutch Booked and made to lose money. An alternative way, which doesn’t require us to assume that all the choices are between monetary payouts, is to construct a series of trades between resources that leaves us with less resources than when we started.
Stated that way, this sounds kinda bad. But then there are things that kind of fit that description, but which we would intuitively think of as good. For instance, some time back I asked:
In response, I was told that
But then I asked that, if we accept this, then what real-life situation does count as an actual circular preference in the VNM sense, given that just about every potential circularity that I can think of is the kind “I prefer A to B at time t1 and B to A at time t2”? And I didn’t get very satisfactory replies.
Intuitively, there are a lot of real-life situations that feel kind of like losing out due to inconsistent preferences, like someone who wants to get into a relationship when he’s single and then wants to be single when he gets into a relationship, but there our actual problem is that the person spends a lot of time being unhappy, rather than with the fact that he makes different choices in different situations. Whereas with the couple, we think that’s fine because they get enjoyment from the “trades”.
The general problem that I’m trying to get at is that in order to hold up VNM rationality as a normative standard, we would need to have a meta-preference: a preference over preferences, stating that it would be better to have preferences that lead to some particular outcomes. The standard Dutch Book example kind of smuggles in that assumption by the way that it talks about money, and thus makes us think that we are in a situation where we are only trying to maximize money and care about nothing else. And if you really are trying to only maximize a single concrete variable or resource and care about nothing else, then you really should try to make sure that your choices follow the VNM axioms. If you run a betting office, then do make sure that nobody can Dutch Book you.
But we don’t have such a clear normative standard for life in general. It would be reasonable to try to construct an argument for why the couple having sex were rational but the person who kept vacillating about being in a relationship was irrational by suggesting that the couple got happiness whereas the other person was unhappy… but we also care about other things than just happiness (or pleasure) and thus aren’t optimizing just for pleasure either. And unless you’re a hedonistic utilitarian, you’re unlikely to say that we should optimize only for pleasure either.
So basically, if you want to say that people should be VNM-rational, then you need to have some specific set of values or goals that you think people should strive towards. If you don’t have that, then VNM-rationality is basically irrelevant aside for the small set of special cases where people really do have a clear explicit goal that’s valued above other things.
I’m not sure I follow in what sense this is a violation of the vNM axioms. A vNM agent has preferences over world-histories; in general one can’t isolate the effect of having an apple vs. having an orange without looking at how that affects the entire future history of the world.
Right, I was trying to say “it prefers an apple to an orange and an orange to an apple in such a way that does violate the axioms”. But I was unsure of what example to actually give of that, since I’m unsure of what real-life situations really would violate the axioms.
The example that comes to mind to show the how the sex thing isn’t a problem is that of a robot car with a goal to drive as many miles as possible. Every day it will burn through all its fuel and fuel up. Right after it fuels up, it will have no desire for further fuel—more fuel simply does not help it go further at this point, and forcing it can be detrimental. Clearly not contradictory
You could have a similar situation with a couple wanting sex iff they haven’t had sex in a day, or wanting an orange if you’ve just eaten an apple but wanting an apple if you’ve just eaten an orange.
To strictly show that something violates vNM axioms, you’d have to show that this behavior (in context) can’t be fulfilling any preferences better than other options that the agent is aware of—or at least be able to argue that the revealed utility function is contrived and unlikely to hold up in other situations (not what the agent “really wants”).
Constantly wanting what one doesn’t have can have this defect. If I keep paying you to switch my apple for your orange and back (without actually eating either), then you have a decent case, if you’re pretty confident I’m not actually fulfilling my desire to troll you ;)
The “want’s a relationship when single” and “wants to be single when not” thing does look like such a violation to me. If you let him flip flop as often as he desires, he’s not going to end up happily endorsing his past actions. If you offered him a pill that would prevent him from flip flopping, he very well may take it. So there’s a contradiction there.
To bring human-specific psychology into it, its not that his inherent desires are contradictory, but that he wants something like “freedom”, which he doesn’t know how to get in a relationship and something like “intimacy”, which he doesn’t know how to get while single. It’s not that he want’s intimacy when single and freedom when not, it’s that he wants both always, but the unfulfilled need is the salient one.
Picture me standing on your left foot. “Oww! Get off my left foot!”. Then I switch to the right “Ahh! Get off my right foot!”. If you’re not very quick and/or the pain is overwhelming, it might take you a few iterations to realize the situation you’re in and to put the pain aside while you think of a way to get me off both feet (intimacy when single/freedom in a relationship). Or if you can’t have that, it’s another challenge to figure out what you want to do about it.
I wouldn’t model you as “just VNM-irrational”, even if your external behaviors are ineffective for everything you might want. I’d model you as “not knowing how to be VNM-rational in presence of strong pain(s)”, and would expect you to start behaving more effectively when shown how.
(and that is what I find, although showing someone how to be more rational is not trivial and “here’s a proof of the inconsistency of your actions now pick a side and stop feeling the desire for the other side” is almost never sufficient. You have to be able to model the specific way that they’re stuck and meet them there)
tl;dr: We’re not VNM-rational because we don’t know how to be, not because it’s not something we’re trying to do.
How do you distinguish his preferences being irrationally inconsistent (he is worse off from entering and leaving relationships repeatedly) from him truly wanting to be in relationships periodically (like how it’s rational to alternate between sleeping and waking rather than always doing one or the other)?
If there’s a pill that can make him stop switching (but doesn’t change his preferences), one of two things will happen: either he’ll never be in a relationship (prevented from entering), or he’ll stay in his current relationship forever (prevented from leaving). I wouldn’t be surprised if he dislikes both of the outcomes and decides not to take the pill.
The pill could instead change his preferences so that he no longer wants to flip-flop, but this argument seems too general—why not just give him a pill that makes him like everything much more than he does now? If my behavior is irrational, I should be able to make myself better off simply by changing my behavior, without having to modify my preferences.
By talking to him. If it’s the latter, he’ll be able to say he prefers flip flopping like it’s just a matter of fact and if you probe into why he likes flip flopping, he’ll either have an answer that makes sense or he’ll talk about it in a way that shows that he is comfortable with not knowing. If it’s the former, he’ll probably say that he doesn’t like flip flopping, and if he doesn’t, it’ll leak signs of bullshit. It’ll come off like he’s trying to convince you of something because he is. And if you probe his answers for inconsistencies he’ll get hostile because he doesn’t want you to.
I’m not sure where you’re going with the “magic pill” hypotheticals, but I agree. The only thing I can think to add is that a lot of times the “winning behaviors” are largely mental and aren’t really available until you understand the situation better.
For example, if you break your foot and can’t get it x-rayed for a day, the right answer might be to just get some writing done—but if you try to force that behavior while you’re suffering, it’s not gonna go well. You have to actually be able to dismiss the pain signal before you have a mental space to write in.
I meant that if someone is behaving irrationally, forcing them to stop that behavior should make them better off. But it seems unlikely to me that forcing him to stay in his current relationship forever, or preventing him from ever entering a relationship (these are the two ways he can be stopped from flip-flopping) actually benefit him.
Forcing anyone to stay in their current relationship forever or forever preventing them from entering a relationship would be quite bad. In order to help him, he’d have to be doing worse than that.
The way to help him would be a bit trickier than that: let him have “good” relationships but not bad. Let him leave “bad” relationships but not good. And then control his mental behaviors so that he’s not allowed to spend time being miserable about his lack of options… (it’s hard to force rationality)
Controlling his mental behaviors would either be changing his preferences or giving him another option. For judging whether he is behaving irrationally, shouldn’t his preferences and set of choices be held fixed?
Relevant question: what does the cognitive science literature on choice-making, preference, and valuation have to say about all this? What mathematical structure actually does model human preferences?
Given that we run on top of neural networks and seem to use some Bayesian algorithms for certain forms of learning (citations available), I currently expect that our choice-making mechanisms might involve conditioning on features or states of our environment at some fundamental level.
My first guess would be that evolution has selected us for circular preferences that our genes money-pump so that we will propagate them. You can’t get off this ride while you’re human.
Is that a challenge?
:-) I mean that if you embody human value, you’ll probably be a money-pumpable entity. Very few humans actually achieve an end to desire while still alive and mentally active.
I’ll take the challenge, then. I was already walking around thinking that the Four Noble Truths of the Buddha are a bunch of depressing bullshit that need to be fixed.
I’ve seen a bunch of different theories backed with varying amounts of experimental data—for instance, this, this and this—but I haven’t looked at them enough to tell which ones seem most correct.
That said, I still don’t remember running into any thorough discussion of what human preferences are, other than just “something that makes us make some choice in some situations”. I mention here that
And I’m a little skeptical of any theory of human preferences that doesn’t attempt to make any such breakdown and only takes a “black box” approach of looking at the outputs of our choice mechanism.
Looks like the relevant textbook came out with an updated edition this year.