Suppose it turned out that humans violate the axioms of VNM rationality (and therefore don’t act like they have utility functions) because there are three valuation systems in the brain that make conflicting valuations, and all three systems contribute to choice.
Utility maximisation is a general framework which is powerful enough to model the actions of any computable agent. The actions of any computable agent—including humans—can be expressed using a utility function. This was spelled out by Dewey in a 2011 paper titled: “Learning What to Value”—in his section about “O-Maximisers”.
Some argue that humans have no utility function. However, this makes little sense: all computable agents have utility functions. The human utility function may not be easy to write down—but that doesn’t mean that it doesn’t exist.
Why would this necessarily be true? Somewhere in mind-design-space is a mind (or AI/algorithm) that confidently asserts A > B, B > C and C > A. (I’m not sufficiently versed in the jargon to know whether this mind would be an “agent”, though—most minds are not goal-seeking in any real sense of the word.)
That mind would have some associated behaviour and that behaviour could be expressed by a utility function (assuming computability—which follows from the Church–Turing–Deutsch principle).
Navel gazing, rushing around in circles, burning money, whatever—all have corresponding utility functions.
Dewey explains why in more detail—if you are prepared to follow the previously-provided link from here.
I’ve taken a look at the paper. If “outcomes” are things like “chose A”, “chose B” or “chose C”, the above mind is simply not an O-maximizer: consider a world with observations “I can choose between A and B/B and C/C and A” (equally likely, independent of any past actions or observations) and actions “take the first offered option” or “take the second offered option” (played for one round, for simplicity, but the argument works fine with multiple rounds); there is no definition of U that yields the described behaviour. (I’m aware that the paper asserts that “any agents [sic] can be written in O-maximizer form”, but note that the paper may simply be wrong. It’s clearly an unfinished draft, and no argument or proof is given.)
If outcomes are things like “chose A given a choice between A and B”, which is not clear to me from the paper, then my mind is indeed an O-maximizer (that is, there is a definition of U such that an O-maximizer produces the same outputs as my mind). However, as I understand it, you have also encoded any cognitive errors in the utility function: if a mind can be Dutch-booked into a undesirable state, the associated O-maximizer will have to act on a U function that values this undesirable state highly if it comes about as a result of being Dutch-booked. (Remember, the O-maximizer maximizes U and behaves like the original mind.) As an additional consideration, most decision/choice theory seems to assume a ranking of outcomes, not (path, outcome) pairs.
I’ve taken a look at the paper. If “outcomes” are things like “chose A”, “chose B” or “chose C”, the above mind is simply not an O-maximizer: consider a world with observations “I can choose between A and B/B and C/C and A” (equally likely, independent of any past actions or observations) and actions “take the first offered option” or “take the second offered option” (played for one round, for simplicity, but the argument works fine with multiple rounds); there is no definition of U that yields the described behaviour.
What?!? You haven’t clearly specified the behaviour of the machine. If you are invoking an uncomputable random number generator to produce an “equally likely” result then you have an uncomputable agent. However, there’s no such thing as an uncomputable random number generator in the real world. So: how is this decision actually being made?
I’m aware that the paper asserts that “any agents [sic] can be written in O-maximizer form”, but note that the paper may simply be wrong. It’s clearly an unfinished draft, and no argument or proof is given.
It applies to any computable agent. That is any agent—assuming that the Church–Turing–Deutsch principle is true.
The argument given is pretty trivial. If you doubt the result, check it—and you should be able to see if it is correct or not fairly easily.
The world is as follows: each observation x_i is one of “the mind can choose between A and B”, “the mind can choose between B and C” or “the mind can choose between C and A” (conveniently encoded as 1, 2 and 3). Independently of any past observations (x_1 and the like) and actions (x_1 and the like), each of these three options is equally likely. This fully specifies a possible world, no?
The mind, then, is as follows: if the last observation is 1 (“A and B”), output “A”; if the last observation is 2 (“B and C”), output “B”; if the last observation is 3 (“C and A”), output “C”. This fully specifies a possible (deterministic, computable) decision procedure, no? (1)
I argue that there is no assignment to U(“A”), U(“B”) and U(“C”) that causes an O-maximizer to produce the same output as the algorithm above. Conversely, there are assignments to U(“1A”), U(“1B”), …, U(“3C”) that cause the O-maximizer to output the same decisions as the above algorithm, but then we have encoded our decision algorithm into the U function used by the O-maximizer (which has its own issues, see my previous post.)
(1) Actually, the definition requires the mind to output something before receiving input. That is a technical detail that can be safely ignored; alternatively, just always output “A” before receiving input.
I argue that there is no assignment to U(“A”), U(“B”) and U(“C”) that causes an O-maximizer to produce the same output as the algorithm above.
...but the domain of a utility function surely includes sensory inputs and remembered past experiences (the state of the agent). You are trying to assign utilities to outputs.
If you try and do that you can’t even encode absolutely elementary preferences with a utility function—such as: I’ve just eaten a peanut butter sandwich, so I would prefer a jam one next.
If that is the only type of utility function you are considering, it is no surprise that you can’t get the theory to work.
The way I know to assign a utility function to an arbitrary agent is to say “I assign what the agent does utility 1, and everything else utility less than one.” Although this “just so” utility function is valid, it doesn’t peek inside the skull—it’s not useful as a model of humans.
What I meant by “how humans make decisions” is a causal model of human decision-making. The reason I wouldn’t call all agents “utility maximizers” is because I want utility maximizers to have a certain causal structure—if you change the probability balance of two options and leave everything else equal, you want it to respond thus. As gwern recently reminded me by linking to that article on Causality, this sort of structure can be tested in experiments.
Although this “just so” utility function is valid, it doesn’t peek inside the skull—it’s not useful as a model of humans.
It’s a model of any computable agent. The point of a utility-based framework capable of modelling any agent is that it allows comparisons between agents of any type. Generality is sometimes a virtue. You can’t easily compare the values of different creatures if you can’t even model those values in the same framework.
The reason I wouldn’t call all agents “utility maximizers” is because I want utility maximizers to have a certain causal structure—if you change the probability balance of two options and leave everything else equal, you want it to respond thus.
Well, you can define your terms however you like—if you explain what you are doing. “Utility” and “maximizer” are ordinary English words, though.
It seems to be impossible to act as though you don’t have a utility function, (as was originally claimed) though. “Utility function” is a perfectly general concept which can be used to model any agent. There may be slightly more concise methods of modelling some agents—that seems to be roughly the concept that you are looking for.
So: it would be possible to say that an agent acts in a manner such that utility maximisation is not the most parsimonious explanation of its behaviour.
Although this “just so” utility function is valid, it doesn’t peek inside the skull—it’s not useful as a model of humans.
It’s a model of any computable agent.
Sorry, replace “model” with “emulation you can use to predict the emulated thing.”
There may be slightly more concise methods of modelling some agents—that seems to be roughly the concept that you are looking for.
I’m talking about looking inside someone’s head and finding the right algorithms running. Rather than “what utility function fits their actions,” I think the point here is “what’s in their skull?”
I’m talking about looking inside someone’s head and finding the right algorithms running. Rather than “what utility function fits their actions,” I think the point here is “what’s in their skull?”
The point made by the O.P. was:
Suppose it turned out that humans violate the axioms of VNM rationality (and therefore don’t act like they have utility functions)
It discussed actions—not brain states. My comments were made in that context.
Er, I don’t think so. To quote from here:
Why would this necessarily be true? Somewhere in mind-design-space is a mind (or AI/algorithm) that confidently asserts A > B, B > C and C > A. (I’m not sufficiently versed in the jargon to know whether this mind would be an “agent”, though—most minds are not goal-seeking in any real sense of the word.)
That mind would have some associated behaviour and that behaviour could be expressed by a utility function (assuming computability—which follows from the Church–Turing–Deutsch principle).
Navel gazing, rushing around in circles, burning money, whatever—all have corresponding utility functions.
Dewey explains why in more detail—if you are prepared to follow the previously-provided link from here.
I’ve taken a look at the paper. If “outcomes” are things like “chose A”, “chose B” or “chose C”, the above mind is simply not an O-maximizer: consider a world with observations “I can choose between A and B/B and C/C and A” (equally likely, independent of any past actions or observations) and actions “take the first offered option” or “take the second offered option” (played for one round, for simplicity, but the argument works fine with multiple rounds); there is no definition of U that yields the described behaviour. (I’m aware that the paper asserts that “any agents [sic] can be written in O-maximizer form”, but note that the paper may simply be wrong. It’s clearly an unfinished draft, and no argument or proof is given.)
If outcomes are things like “chose A given a choice between A and B”, which is not clear to me from the paper, then my mind is indeed an O-maximizer (that is, there is a definition of U such that an O-maximizer produces the same outputs as my mind). However, as I understand it, you have also encoded any cognitive errors in the utility function: if a mind can be Dutch-booked into a undesirable state, the associated O-maximizer will have to act on a U function that values this undesirable state highly if it comes about as a result of being Dutch-booked. (Remember, the O-maximizer maximizes U and behaves like the original mind.) As an additional consideration, most decision/choice theory seems to assume a ranking of outcomes, not (path, outcome) pairs.
What?!? You haven’t clearly specified the behaviour of the machine. If you are invoking an uncomputable random number generator to produce an “equally likely” result then you have an uncomputable agent. However, there’s no such thing as an uncomputable random number generator in the real world. So: how is this decision actually being made?
It applies to any computable agent. That is any agent—assuming that the Church–Turing–Deutsch principle is true.
The argument given is pretty trivial. If you doubt the result, check it—and you should be able to see if it is correct or not fairly easily.
The world is as follows: each observation x_i is one of “the mind can choose between A and B”, “the mind can choose between B and C” or “the mind can choose between C and A” (conveniently encoded as 1, 2 and 3). Independently of any past observations (x_1 and the like) and actions (x_1 and the like), each of these three options is equally likely. This fully specifies a possible world, no?
The mind, then, is as follows: if the last observation is 1 (“A and B”), output “A”; if the last observation is 2 (“B and C”), output “B”; if the last observation is 3 (“C and A”), output “C”. This fully specifies a possible (deterministic, computable) decision procedure, no? (1)
I argue that there is no assignment to U(“A”), U(“B”) and U(“C”) that causes an O-maximizer to produce the same output as the algorithm above. Conversely, there are assignments to U(“1A”), U(“1B”), …, U(“3C”) that cause the O-maximizer to output the same decisions as the above algorithm, but then we have encoded our decision algorithm into the U function used by the O-maximizer (which has its own issues, see my previous post.)
(1) Actually, the definition requires the mind to output something before receiving input. That is a technical detail that can be safely ignored; alternatively, just always output “A” before receiving input.
...but the domain of a utility function surely includes sensory inputs and remembered past experiences (the state of the agent). You are trying to assign utilities to outputs.
If you try and do that you can’t even encode absolutely elementary preferences with a utility function—such as: I’ve just eaten a peanut butter sandwich, so I would prefer a jam one next.
If that is the only type of utility function you are considering, it is no surprise that you can’t get the theory to work.
The point is about how humans make decisions, not about what decisions humans make.
Er, what are you talking about? Did you not understand what was wrong with Luke’s sentence? Or what are you trying to say?
The way I know to assign a utility function to an arbitrary agent is to say “I assign what the agent does utility 1, and everything else utility less than one.” Although this “just so” utility function is valid, it doesn’t peek inside the skull—it’s not useful as a model of humans.
What I meant by “how humans make decisions” is a causal model of human decision-making. The reason I wouldn’t call all agents “utility maximizers” is because I want utility maximizers to have a certain causal structure—if you change the probability balance of two options and leave everything else equal, you want it to respond thus. As gwern recently reminded me by linking to that article on Causality, this sort of structure can be tested in experiments.
It’s a model of any computable agent. The point of a utility-based framework capable of modelling any agent is that it allows comparisons between agents of any type. Generality is sometimes a virtue. You can’t easily compare the values of different creatures if you can’t even model those values in the same framework.
Well, you can define your terms however you like—if you explain what you are doing. “Utility” and “maximizer” are ordinary English words, though.
It seems to be impossible to act as though you don’t have a utility function, (as was originally claimed) though. “Utility function” is a perfectly general concept which can be used to model any agent. There may be slightly more concise methods of modelling some agents—that seems to be roughly the concept that you are looking for.
So: it would be possible to say that an agent acts in a manner such that utility maximisation is not the most parsimonious explanation of its behaviour.
Sorry, replace “model” with “emulation you can use to predict the emulated thing.”
I’m talking about looking inside someone’s head and finding the right algorithms running. Rather than “what utility function fits their actions,” I think the point here is “what’s in their skull?”
The point made by the O.P. was:
It discussed actions—not brain states. My comments were made in that context.