It seems worth reflecting on the fact that the point of the foundational LW material discussing utility functions was to make people better at reasoning about AI behavior and not about human behavior.
I think part of Eliezer’s point was also to introduce decision theory as an ideal for human rationality. (See http://lesswrong.com/lw/my/the_allais_paradox/ for example.) Without talking about utility functions, we can’t talk about expected utility maximization, so we can’t define what it means to be ideally rational in the instrumental sense (and we also can’t justify Bayesian epistemology based on decision theory).
So I agree with the problem stated here, but “let’s stop talking about utility functions” can’t be the right solution. Instead we need to emphasize more that having the wrong values is often worse than being irrational, so until we know how to obtain or derive utility functions that aren’t wrong, we shouldn’t try to act as if we have utility functions.
The trouble is the people who read the Sequences and went “EY said it, it’s probably right, I’ll internalise it.” This is an actual hazard around here. (Even Eliezer can’t make people think, rather than just believe in thinking.)
Yes, decision theory has been floated as a normative standard for human rationality. The trouble is that the standard is bogus. Conformity to the full set of axioms is not a rational requirement. The Allais Paradox and the Ellsberg Paradox are cases in point. Plenty of apparently very intelligent and rational people make decisions that violate the axioms, even when shown how their decisions violate the VNM axioms. I tentatively conclude that the problem lies in the axioms, rather than these decision makers. In particular, the Independence of “Irrelevant” Alternatives and some strong ordering assumptions both look problematic. Teddy Seidenfeld has a good paper on the ordering assumptions.
Without talking about utility functions, we can’t talk about expected utility maximization, so we can’t define what it means to be ideally rational in the instrumental sense
I like this explanation of why utility-maximization matters for Eliezer’s overarching argument. I hadn’t noticed that before.
But it seems like utility functions are an unnecessarily strong assumption here. If I understand right, expected utility maximization and related theorems imply that if you have a complete preference over outcomes, and have probabilities that tell you how decisions influence outcomes, you have implicit preferences over decisions.
But even if you have only partial information about outcomes and partial preferences, you still have some induced ordering of the possible actions. We lose the ability to show that there is always an optimal ‘rational’ decision, but we can still talk about instances of irrational decision-making.
It’s not obvious to me that Qiaochu would endorse utility functions as a standard for “ideal rationality”. I, for one, do not.
Talking about utility functions can be useful if one believes any of the following about ideal rationality, as a concrete example of what one means if nothing else.
An ideally rational agent uses one of the standard decision theories (vNM, EDT, CDT, etc.)
An ideally rational agent does EU maximization.
An ideally rational agent is consequentialist.
An ideally rational agent, when evaluating the consequences of its actions, divides up the domain of evaluation into two or more parts, evaluates them separately, and then adds their values together. (For example, for an EU maximizer, the “parts” are possible outcomes or possible world-histories. For a utilitarian, the “parts” are individual persons within each world.)
An ideally rational agent has values/preferences that are (or can be) represented by a clearly defined mathematical object.
I guess when you say you don’t “endorse utility functions” you mean that you don’t endorse 1 or 2. Do you endorse any of the others, and if so what would you use instead of utility functions to illustrate what you mean?
It’s hard for me to know that 4 and 5 really mean since they are so abstract. I definitely don’t endorse 1 or 2 and I’m pretty sure I don’t endorse 4 either (integrating over uncertainty in what you meant). I’m uncertain about 3; it seems plausible but far from clear. I’m certainly not consequentialist and don’t want to be, but maybe I would want to be in some utopian future. Again, I’m not really sure what you mean by 5, it seems almost tautological since everything is a mathematical object.
Even if you don’t think it’s the ideal, utility based decision theory it does give us insights that I don’t think you can naturally pick up from anywhere else that we’ve discovered yet.
I think part of Eliezer’s point was also to introduce decision theory as an ideal for human rationality. (See http://lesswrong.com/lw/my/the_allais_paradox/ for example.) Without talking about utility functions, we can’t talk about expected utility maximization, so we can’t define what it means to be ideally rational in the instrumental sense (and we also can’t justify Bayesian epistemology based on decision theory).
So I agree with the problem stated here, but “let’s stop talking about utility functions” can’t be the right solution. Instead we need to emphasize more that having the wrong values is often worse than being irrational, so until we know how to obtain or derive utility functions that aren’t wrong, we shouldn’t try to act as if we have utility functions.
The trouble is the people who read the Sequences and went “EY said it, it’s probably right, I’ll internalise it.” This is an actual hazard around here. (Even Eliezer can’t make people think, rather than just believe in thinking.)
Yes, decision theory has been floated as a normative standard for human rationality. The trouble is that the standard is bogus. Conformity to the full set of axioms is not a rational requirement. The Allais Paradox and the Ellsberg Paradox are cases in point. Plenty of apparently very intelligent and rational people make decisions that violate the axioms, even when shown how their decisions violate the VNM axioms. I tentatively conclude that the problem lies in the axioms, rather than these decision makers. In particular, the Independence of “Irrelevant” Alternatives and some strong ordering assumptions both look problematic. Teddy Seidenfeld has a good paper on the ordering assumptions.
I like this explanation of why utility-maximization matters for Eliezer’s overarching argument. I hadn’t noticed that before.
But it seems like utility functions are an unnecessarily strong assumption here. If I understand right, expected utility maximization and related theorems imply that if you have a complete preference over outcomes, and have probabilities that tell you how decisions influence outcomes, you have implicit preferences over decisions.
But even if you have only partial information about outcomes and partial preferences, you still have some induced ordering of the possible actions. We lose the ability to show that there is always an optimal ‘rational’ decision, but we can still talk about instances of irrational decision-making.
It’s not obvious to me that Qiaochu would endorse utility functions as a standard for “ideal rationality”. I, for one, do not.
Talking about utility functions can be useful if one believes any of the following about ideal rationality, as a concrete example of what one means if nothing else.
An ideally rational agent uses one of the standard decision theories (vNM, EDT, CDT, etc.)
An ideally rational agent does EU maximization.
An ideally rational agent is consequentialist.
An ideally rational agent, when evaluating the consequences of its actions, divides up the domain of evaluation into two or more parts, evaluates them separately, and then adds their values together. (For example, for an EU maximizer, the “parts” are possible outcomes or possible world-histories. For a utilitarian, the “parts” are individual persons within each world.)
An ideally rational agent has values/preferences that are (or can be) represented by a clearly defined mathematical object.
I guess when you say you don’t “endorse utility functions” you mean that you don’t endorse 1 or 2. Do you endorse any of the others, and if so what would you use instead of utility functions to illustrate what you mean?
It’s hard for me to know that 4 and 5 really mean since they are so abstract. I definitely don’t endorse 1 or 2 and I’m pretty sure I don’t endorse 4 either (integrating over uncertainty in what you meant). I’m uncertain about 3; it seems plausible but far from clear. I’m certainly not consequentialist and don’t want to be, but maybe I would want to be in some utopian future. Again, I’m not really sure what you mean by 5, it seems almost tautological since everything is a mathematical object.
Even if you don’t think it’s the ideal, utility based decision theory it does give us insights that I don’t think you can naturally pick up from anywhere else that we’ve discovered yet.