(I realize Eliezer is familiar with the problems with taking average utility; I write this for those following the conversation.)
So, if we are to choose between supporting a population of 1,000,000 people with a utility of 10, or 1 person with a utility of 11, we should choose the latter? If someone’s children are going to be born into below-average circumstances, it would be better for us to prevent them from having children?
(I know that you spoke of all living people; but we need a definition of rationality that addresses changes in population.)
Inequitable distributions of utility are as good as equitable distributions of utility? You have no preference between 1 person with a utility of 100, and 9 people with utilities of 0, versus 10 people with utilities of 10? (Do not invoke economics to claim that inequitable distributions of utility are necessary for productivity. This has nothing to do with that.)
Ursula LeGuin wrote a short story about this, called “The ones who walk away from Omelas”, which won the Hugo in 1974. (I’m not endorsing it; merely noting it.)
You don’t interpret “utility” the same way others here do, just like the word “happiness”. Our utility inherently includes terms for things like inequity. What you are using the word “utility” here for would be better described as “happiness”.
Since your title said “maximizing expected utility is wrong” I assumed that the term “average” was to be taken in the sense of “average over probabilities”, but yes, in a Big and possibly Infinite World I tend toward average utilitarianism.
You don’t interpret “utility” the same way others here do, just like the word “happiness”. Our utility inherently includes terms for things like inequity. What you are using the word “utility” here for would be better described as “happiness”.
We had the happiness discussion already. I’m using the same utility-happiness distinction now as then.
(You’re doing that “speaking for everyone” thing again. Also, what you would call “speaking for me”, and misinterpreting me. But that’s okay. I expect that to happen in conversations.)
Our utility inherently includes terms for things like inequity.
The little-u u(situation) can include terms for inequity.
The big-U U(lottery of situations) can’t, if you’re an expected utility maximizer. You are constrained to aggregate over different outcomes by averaging.
Since the von Neumann-Morgenstern theorem indicates that averaging is necessary in order to avoid violating their reasonable-seeming axioms of utility, my question is then whether it is inconsistent to use expected utility over possible outcomes, and NOT use expected utility across people.
Since you do both, that’s perfectly consistent. The question is whether anything else makes sense in light of the von Neumann-Morgenstern theorem.
If you maximize expected utility, that means that an action that results in utility 101 for one future you in one possible world, and utility 0 for 9 future yous in 9 equally-likely possible worlds; is preferable to an action that results in utility 10 for all 10 future yous. That is very similar to saying that you would rather give utilty 101 to 1 person and utility 0 to 9 other people, than utility 10 to 10 people.
If your utility function were defined over all possible worlds, you would just say “maximize utility” instead of “maximize expected utility”.
I disagree: that’s only the case if you have perfect knowledge.
Case A: I’m wondering whether to flip the switch of my machine. The machine causes a chrono-synclastic infundibulum, which is a physical phenomenon that has a 50% chance of causing a lot of awesomeness (+100 utility), and a 50% chance of blowing up my town (-50 utility).
Case B: I’m wondering whether to flip the switch of my machine, a friendly AI I just programmed. I don’t know whether I programmed it right, if I did it will bring forth an awesome future (+100 utility), if I didn’t it will try to enslave mankind (-50 utility). I estimate that my program has 50% chances of being right.
Both cases are different, and if you have a utility function that’s defined over all possible future words (that just takes the average), you could say that flipping the switch in the first case has utility of +50, and in the second case, expected utility of +50 (actually, utility of +100 or −50, but you don’t know which).
Phil, this is something eerie, totally different from the standard von Neumann-Morgenstern expected utility over the world histories, which is what people usually refer to when talking about the ideal view on the expected utility maximization. Why do you construct this particular preference order? What do you answer to the standard view?
I don’t understand the question. Did I define a preference order? I thought I was just pointing out an unspoken assumption. What is the difference between what I have described as maximizing expected utility, and the standard view?
The following passage is very strange, it shows either lack of understanding, or some twisted terminology.
A utility measure discounts for inequities within any single possible outcome. It does not discount for utilities across the different possible outcomes. It can’t, because utility functions are defined over a single world, not over the set of all possible worlds. If your utility function were defined over all possible worlds, you would just say “maximize utility” instead of “maximize expected utility”.
It shows twisted terminology. I rewrote the main post to try to fix it.
I’d like to delete the whole post in shame, but I’m still confused as to whether we can be expected utility maximizers without being average utilitarianists.
I’ve thought about this a bit more, and I’m back to the intuition that you’re mixing up different concepts of “utility” somewhere, but I can’t make that notion any more precise. You seem to be suggesting that certain seemingly plausible preferences cannot be properly expressed as utility functions. Can you give a stripped-down, “single-player” example of this that doesn’t involve other people or selves?
You seem to be suggesting that certain seemingly plausible preferences cannot be properly expressed as utility functions.
Here’s a restatement:
We have a utility function u(outcome) that gives a utility for one possible outcome.
We have a utility function U(lottery) that gives a utility for a probability distribution over all possible outcomes.
The von Neumann-Morgenstern theorem indicates that the only reasonable form for U is to calculate the expected value of u(outcome) over all possible outcomes.
This means that your utility function U is indifferent with regard to whether the distribution of utility is equitable among your future selves. Giving one future self u=10 and another u=0 is equally as good as giving one u=5 and another u=5.
This is the same sort of ethical judgement that an average utilitarian makes when they say that, to calculate social good, we should calculate the average utility of the population.
Therefore, I think that the von Neumann-Morgenstern theorem does not prove, but provides very strong reasons for thinking, that average utilitarianism is correct.
And yet, average utilitarianism asserts that equity of utility, even among equals has no utility. This is shocking.
If you want a more equitable distribution of utility among future selves, then your utility function u(outcome) may be a different function than you thought it was; e.g. the log of the function you thought it was.
More generally, if u is the function that you thought was your utility function, and f is any monotonically increasing function on the reals with f″ < 0, then by Jensen’s inequality, an expected f″(u)-maximizer would prefer to distribute u-utility equitably among its future selves.
Exactly. (I didn’t realize the comments were continuing down here and made the essentially same point here after Phil amended the post.)
The interesting point that Phil raises is whether there’s any reason to have a particular risk preference with respect to u. I’m not sure that the analogy between being inequality averse amongst possible “me”s and and inequality averse amongst actual others gets much traction once we remember that probability is in the mind. But it’s an interesting question nonetheless.
Allais, in particular argued that any form of risk preference over u should be allowable, and Broome finds this view “very plausible”. All of which seems to make rational decision-making under uncertainty much more difficult, particularly as it’s far from obvious that we have intuitive access to these risk preferences. (I certainly don’t have intuitive access to mine.)
P.S. I assume you mean f(u)-maximizer rather than f″(u)-maximizer?
Yes—and then the f(u)-maximizer is not maximizing expected utility! Maximizing expected utility requires not wanting equitable distribution of utility among future selves.
This is the same sort of ethical judgement that an average utilitarian makes when they say that, to calculate social good, we should calculate the average utility of the population.
Nope. You can have u(10 people alive) = −10 and u(only 1 person is alive)=100 or u(1 person is OK and another suffers)=100 and u(2 people are OK)=-10.
I objected to drawing the analogy, and gave the examples that show where the analogy breaks. Utility over specific outcomes values the whole world, with all people in it, together. Alternative possibilities for the whole world figuring into the expected utility calculation are not at all the same as different people. People that the average utilitarianism talks about are not from the alternative worlds, and they do not each constitute the whole world, the whole outcome. This is a completely separate argument, having only surface similarity to the expected utility computation.
We have a utility function u(outcome) that gives a utility for one possible outcome.
We have a utility function U(lottery) that gives a utility for a probability distribution over all possible outcomes.
The von Neumann-Morgenstern theorem indicates that the only reasonable form for U is to calculate the expected value of u(outcome) over all possible outcomes.
I’m with you so far.
This means that your utility function U is indifferent with regard to whether the distribution of utility is equitable among your future selves. Giving one future self u=10 and another u=0 is equally as good as giving one u=5 and another u=5.
What do you mean by “distribute utility to your future selves”? You can value certain circumstances involving future selves higher than others, but when you speak of “their utility” you’re talking about a completely different thing than the term u in your current calculation. u already completely accounts for how much they value their situation and how much you care whether or not they value it.
This is the same sort of ethical judgement that an average utilitarian makes when they say that, to calculate social good, we should calculate the average utility of the population.
I don’t see how this at all makes the case for adopting average utilitarianism as a value framework, but I think I’m missing the connection you’re trying to draw.
I’d hate to see it go. I think you’ve raised a really interesting point, despite not communicating it clearly (not that I can probably even verbalize it yet). Once I got your drift it confused the hell out of me, in a good way.
Assuming I’m correct that it was basically unrelated, I think your previous talk of “happiness vs utility” might have primed a few folks to assume the worst here.
Phil, you’re making a claim that what others say about utility (i.e. that it’s good to maximize its expectation) is wrong. But it’s only on your idiosyncratic definition of utility that your argument has any traction.
You are free to use words any way you want (even if I personally find your usage frustrating at times). But you are not free to redefine others’ terms to generate an artificial problem that isn’t really there.
The injunction to “maximize expected utility” is entirely capable of incorporating your concerns. It can be “inequality-averse” if you want, simply by making it a concave function of experienced utility.
The injunction to “maximize expected utility” is entirely capable of incorporating your concerns. It can be “inequality-averse” if you want, simply by making it a concave function of experienced utility
No. I’ve said this 3 times already, including in the very comment that you are replying to. The utility function is not defined across all possible outcomes. A utility function is defined over a single outcome; it evaluates a single outcome. It can discount inequalities within that outcome. It cannot discount across possible worlds. If it operated across all possible worlds, all you would say is “maximize utility”. The fact that you use the word “expected” means “average over all possible outcomes”. That is what “expected” means. It is a mathematical term whose meaning is already established.
You can safely ignore my previous reply, I think I finally see what you’re saying. Not sure what to make of it yet, but I was definitely misinterpreting you.
Repeating your definition of a utility function over and over again doesn’t oblige anybody else to use it. In particular, it doesn’t oblige all those people who have argued for expected utility maximization in the past to have adopted it before you tried to force it on them.
A von Neumann-Morgenstern utility function (which is what people are supposed to maximize the expectation of) is a representation of a set of consistent preferences over gambles. That is all it is. If your proposal results in a set of consistent preferences over gambles (I see no particular reason for it not to, but I could be wrong) then it corresponds to expected utility maximization for some utility function. If it doesn’t, then either it is inconsistent, or you have a beef with the axioms that runs deeper than an analogy to average utilitarianism.
If you don’t prefer 10% chance of 101 utilons to 100% chance of 10, then you can rescale your utility function (in a non-affine manner). I bet you’re thinking of 101 as “barely more than 10 times as much” of something that faces diminishing returns. Such diminishing returns should already be accounted for in your utility function.
I bet you’re thinking of 101 as “barely more than 10 times as much” of something that faces diminishing returns.
No. I’ve explained this in several of the other comments. That’s why I used the term “utility function”, to indicate that diminishing returns are already taken into account.
It can’t, because utility functions are defined over a single world, not over the set of all possible worlds. If your utility function were defined over all possible worlds, you would just say “maximize utility” instead of “maximize expected utility”.
This doesn’t sound right to me. Assuming “world” means “world at time t”, a utility function at the very least has type (World → Utilons). It maps a single world to a single utility measure, but it’s still defined over all worlds, the same way that (+3) is defined over all integers. If it was only defined for a single world it wouldn’t really be much of a function, it’d be a constant.
We use expected utility due to uncertainty. If we had perfect information, we could maximize utility by searching over all action sequences, computing utility for each resulting world, and returning the sequence with the highest total utility.
If you maximize expected utility, that means that an action that results in utility 101 for one future you in one possible world, and utility 0 for 9 future yous in 9 equally-likely possible worlds
I think this illustrates the problem with your definition. The utility you’re maximizing is not the same as the “utility 101 for one future you”. You first have to map future you’s utility to just plain utility for any of this to make sense.
It maps a single world to a single utility measure, but it’s still defined over all worlds,
I meant “the domain of a utility function is a single world.”
However, it turns out that the standard terminology includes both utility functions over a single world (“outcome”), and a big utility function over all possible worlds (“lottery”).
My question/observation is still the same as it was, but my misuse of the terminology has mangled this whole thread.
The reason why an inequitable distribution of money is problematic is that money has diminishing marginal utility; so if a millionaire gives $1000 to a poor person, the poor person gains more than the millionaire loses.
If your instincts are telling you that an inequitable distribution of utility is bad, are you sure you’re not falling into the “diminishing marginal utility of utility” error that people have been empirically shown to exhibit? (can’t find link now, sorry, I saw it here).
The reason why an inequitable distribution of money is problematic is that money has diminishing marginal utility; so if a millionaire gives $1000 to a poor person, the poor person gains more than the millionaire loses.
Perhaps it would help if you gave a specific example of an action that (a) follows from average utilitarianism as you understand it, and (b) you believe most people would find reprehensible?
The standard answer is killing a person with below-average well-being*, assuming no further consequences follow from this. This assumes dying has zero disutility, however.
*The term “experienced utility” seems to be producing a lot of confusion. Utility is a decision-theoretic construction only. Humans, as is, don’t have utility functions.
Yes, I’m surprised that it’s average rather than total utility is being measured. All other things being equal, twice as many people is twice as good to me.
The standard answer is killing a person with below-average well-being*, assuming no further consequences follow from this. This assumes dying has zero disutility, however.
Dead people presumably count as zero utility. I was rather frightened before I saw that—if you only count living people, then you’d be willing to kill people for the crime of not being sufficiently happy or fulfilled.
This sentence doesn’t really mean much. A dead person doesn’t have preferences or utility (zero or otherwise) when dead any more than a rock does, a dead person had preferences and utility when alive. The death of a living person (who preferred to live) reduces average utility because the living person preferred to not die, and that preference is violated!
you’d be willing to kill people for the crime of not being sufficiently happy or fulfilled
I support the right to euthanasia for people who truly prefer to be killed, e.g. because they suffer from terminal painful diseases. Do you oppose it?
Perhaps you should see it more clearly if you think of it as maximizing the average preference utility across the timeline, rather than the average utility at a single point in time.
The death of a living person (who preferred to live) reduces average utility because the living person preferred to not die, and that preference is violated!
But after the fact, they are not alive, so they do not impact the average utility across all living things, so you are increasing the average utility across all living things.
Here’s what I mean, roughly expressed. Two possible timelines. (A)
2010 Alice loves her life (she wants to live with continued life being a preference satisfaction of 7 per year). Bob merely likes his life (he wants to live with continued life being a preference satisfaction of 6 per year).
2011 Alice and 2011 Bob both alive as before.
2012 Alice and 2012 Bob both alive as before.
2013 Alice and 2013 Bob both alive as before.
2010 Bob wants 2010 Bob to exist, 2011 Bob to exist, 2012 Bob to exist, 2013 Bob to exist. 2010 Alice wants 2010 Alice to exist, 2011 Alice to exist, 2012 Alice to exist, 2013 Alice to exist.
2010 average utility is therefore (4x7 + 4x6) / 2= 26 and that also remains the average for the whole timeline.
(B)
2010 Alice and Bob same as before.
2011 Alice is alive. Bob has just been killed.
2012 Alice alive as before.
2013 Alice alive as before.
2010 average utility is: 4x7 +6 = (4x7 + 1x6)/2 = 17 2011 average utility is: 4x7 = 28 2012 average utility is: 4x7 = 28 2013 average utility is: 4x7 = 28
So Bob’s death increased the average utility indicated in the preferences of a single year. But average utilty across the timeline is now (28 + 6 + 28 + 28 + 28) / 5 = 23.6
In short the average utility of the timeline as a whole is decreased by taking out Bob.
You are averaging based on the population at the start of the experiment. In essence, you are counting dead people in your average, like Eliezer’s offhanded comment implied he would. Also, you are summing over the population rather than averaging.
Correcting those discrepancies, we would see (ua ⇒ “utils average”; murder happening New Year’s Day 2011):
Now, let’s say we are using procreation instead of murder as the interesting behavior. Let’s say each act of procreation reduces the average utility by 1, and it starts at 100 at the beginning of the experiment, with an initial population of 10.
In the first year, we can decrease the average utility by 10 in order to add one human with 99 utility. When do we stop adding humans? Well, it’s clear that the average utility in this contrived example is equal to 110 minus the total population, and the total utility is equal to the average times the population size. If we have 60 people, that means our average utility is 50, with a total of 3,000 utils. Three times as good for everyone, except half as good for our original ten people.
We maximize the utility at a population of 55 in this example (and 55 average utility) -- but that’s because we can’t add new people very efficiently. If we had a very efficient way of adding more people, we’d end up with the average utility being just barely better than death, but we’d make up for it in volume. That’s what you are suggesting we do.
That also isn’t a universe I want to live in. Eliezer is suggesting we count dead people in our averages, nothing more. That’s sufficient to go from kill-almost-all-humans to something we can maybe live with. (Of course, if we counted their preferences, that would be a conservatizing force that we could never get rid of, which is similarly worrying, albeit not as much so. In the worst case, we could use an expanding immortal population to counter it. Still-living people can change their preferences.)
You are averaging based on the population at the start of the experiment. In essence, you are counting dead people in your average, like Eliezer’s offhanded comment implied he would
I consider every moment of living experience as of equal weight. You may call that “counting dead people” if you want, but that’s only because when considering the entire timeline I consider every living moment—given a single timeline, there’s no living people vs dead people, there’s just people living in different times. If you calculate the global population it doesn’t matter what country you live in—if you calculate the utility of a fixed timeline, it doesn’t matter what time you live in.
But the main thing I’m not sure you get is that I believe preferences are valid also when concerning the future, not just when concerning the present.
If 2014 Carl wants the state of the world to be X in 2024, that’s still a preference to be counted, even if Carl ends up dead in the meantime. That Carl severely does NOT want to be dead in 2024, means that there’s a heavy disutility penalty for the 2014 function of his utility if he ends up nonetheless dead in 2024.
Of course, if we counted their preferences, that would be a conservatizing force that we could never get rid of
If e.g. someone wants to be buried at sea because he loves the sea, I consider it good that we bury them at sea. But if someone wants to be buried at sea only because he believes such a ritual is necessary for his soul to be resurrected by God Poseidon, his preference is dependent on false beliefs—it doesn’t represent true terminal values; and that’s the ones I’m concerned about.
If conservatism is e.g. motivated by either wrong epistemic beliefs, or by fear, rather than true different terminal values, it should likewise not modify our actions, if we’re acting from an epistemically superior position (we know what they didn’t).
when considering the entire timeline I consider every living moment—given a single timeline, there’s no living people vs dead people, there’s just people living in different times. If you calculate the global population it doesn’t matter what country you live in—if you calculate the utility of a fixed timeline, it doesn’t matter what time you live in.
That’s an ingenious fix, but when I think about it I’m not sure it works. The problem is that although you are calculating the utility integrated over the timeline, the values that you are integrating are still based on a particular moment. In other words, calculating the utility of the 2014-2024 timeline by 2014 preferences might not produce the same result as calculating the utility of the 2014-2024 timeline by 2024 preferences. Worse yet, if you’re comparing two timelines and the two timelines have different 2024s in them, and you try to compare them by 2024 preferences, which timeline’s 2024 preferences do you use?
For instance, consider
timeline A: Carl is alive in 2014 and is killed soon afterwards, but two new people are born who are alive in 2024.
timeline B: Carl is alive in 2014 and in 2024, but the two people from A never existed.
If you compare the timelines by Carl’s 2014 preferences or Carl’s timeline B 2024 preferences, timeline B is better, because timeline B has a lot of utility integrated over Carl’s life.
If you compare the timelines by the other people’s timeline A 2024 preferences, timeline A is better.
It’s tempting to try to fix this argument by saying that rather than using preferences at a particular moment, you will use preferences integrated over the timeline, but if you do that in the obvious way (by weighting the preferences according to the person-hours spent with that preference), then killing someone early reduces the contribution of their preference to the integrated utility, causing a problem similar to the original one.
you’d be willing to kill people for the crime of not being sufficiently happy or fulfilled
I support the right to euthanasia for people who truly prefer to be killed, e.g. because they suffer from terminal painful diseases. Do you oppose it?
I believe that what dhasenan was getting at is that without the assumption that a dead person has 0 utility, you would be willing to kill people who are happy (positive utility), but just not as happy as they could be. I’m not sure how exactly this would go mathematically, but the point is that killing a +utility person being a reduction in utility is a vital axiom
It’s not that they could be happier. Rather, if the average happiness is greater than my happiness, the average happiness in the population will be increased if I die (assuming the other effects of a person dying are minimal or sufficiently mitigated).
but the point is that killing a +utility person being a reduction in utility is a vital axiom
I don’t know if we need have it as an axiom rather than this being a natural consequence of happy people preferring not to be killed, and of us likewise preferring not to kill them, and of pretty much everyone preferring their continued lives to their deaths… The good of preference utilitarianism is that it takes all these preferences as an input.
If preference average utilitarianism nonetheless leads to such an abominable conclusion, I’ll choose to abandon preference average utilitarianism, considering it a failed/misguided attempt at describing my sense of morality—but I’m not certain it needs lead to such a conclusion at all.
If I agreed, I’d be extremely curious as to what the average utility for all people across the multiverse actually is. (Is it dominated by people with extremely short lifespans, because they use so little computing power in a 4D sense?)
Why? Eliezer said he wanted to maximize the average utility of all people, then said that average utility was 67. Now he faces the difficult task of maximizing 67. Or maybe maximizing the utility of people who share a planet with him, at the expense of other people in existence, so the average stays 67. Am I missing something? :-)
Excuse me, what’s “average utility”? How do you compare utils of different people? Don’t say you’re doing it through the lens of your own utility function—this is begging the question.
They’re easy to help if their utility functions included terms that outlived them, e.g. “world peace forever”. But it still feels somehow wrong to include them in the calculation, because this will necessarily be at the expense of living and future people.
In my non-professional capacity, when I try to help others, I’m doing so to optimize my utility function over them: I want them to be happy, and only living people can fulfill this aspect of my utility function. It’s in this sense that I say “we should” meaning “altruists should do the analogue for their own utility functions”.
We should maximize average utility across all living people.
(Actually all people, but dead people are hard to help.)
As is well known, I have a poor model of Eliezer.
(I realize Eliezer is familiar with the problems with taking average utility; I write this for those following the conversation.)So, if we are to choose between supporting a population of 1,000,000 people with a utility of 10, or 1 person with a utility of 11, we should choose the latter? If someone’s children are going to be born into below-average circumstances, it would be better for us to prevent them from having children?
(I know that you spoke of all living people; but we need a definition of rationality that addresses changes in population.)
Inequitable distributions of utility are as good as equitable distributions of utility? You have no preference between 1 person with a utility of 100, and 9 people with utilities of 0, versus 10 people with utilities of 10? (Do not invoke economics to claim that inequitable distributions of utility are necessary for productivity. This has nothing to do with that.)
Ursula LeGuin wrote a short story about this, called “The ones who walk away from Omelas”, which won the Hugo in 1974. (I’m not endorsing it; merely noting it.)
You don’t interpret “utility” the same way others here do, just like the word “happiness”. Our utility inherently includes terms for things like inequity. What you are using the word “utility” here for would be better described as “happiness”.
Since your title said “maximizing expected utility is wrong” I assumed that the term “average” was to be taken in the sense of “average over probabilities”, but yes, in a Big and possibly Infinite World I tend toward average utilitarianism.
We had the happiness discussion already. I’m using the same utility-happiness distinction now as then.
(You’re doing that “speaking for everyone” thing again. Also, what you would call “speaking for me”, and misinterpreting me. But that’s okay. I expect that to happen in conversations.)
The little-u u(situation) can include terms for inequity. The big-U U(lottery of situations) can’t, if you’re an expected utility maximizer. You are constrained to aggregate over different outcomes by averaging.
Since the von Neumann-Morgenstern theorem indicates that averaging is necessary in order to avoid violating their reasonable-seeming axioms of utility, my question is then whether it is inconsistent to use expected utility over possible outcomes, and NOT use expected utility across people.
Since you do both, that’s perfectly consistent. The question is whether anything else makes sense in light of the von Neumann-Morgenstern theorem.
If you maximize expected utility, that means that an action that results in utility 101 for one future you in one possible world, and utility 0 for 9 future yous in 9 equally-likely possible worlds; is preferable to an action that results in utility 10 for all 10 future yous. That is very similar to saying that you would rather give utilty 101 to 1 person and utility 0 to 9 other people, than utility 10 to 10 people.
I disagree: that’s only the case if you have perfect knowledge.
Case A: I’m wondering whether to flip the switch of my machine. The machine causes a chrono-synclastic infundibulum, which is a physical phenomenon that has a 50% chance of causing a lot of awesomeness (+100 utility), and a 50% chance of blowing up my town (-50 utility).
Case B: I’m wondering whether to flip the switch of my machine, a friendly AI I just programmed. I don’t know whether I programmed it right, if I did it will bring forth an awesome future (+100 utility), if I didn’t it will try to enslave mankind (-50 utility). I estimate that my program has 50% chances of being right.
Both cases are different, and if you have a utility function that’s defined over all possible future words (that just takes the average), you could say that flipping the switch in the first case has utility of +50, and in the second case, expected utility of +50 (actually, utility of +100 or −50, but you don’t know which).
Phil, this is something eerie, totally different from the standard von Neumann-Morgenstern expected utility over the world histories, which is what people usually refer to when talking about the ideal view on the expected utility maximization. Why do you construct this particular preference order? What do you answer to the standard view?
I don’t understand the question. Did I define a preference order? I thought I was just pointing out an unspoken assumption. What is the difference between what I have described as maximizing expected utility, and the standard view?
The following passage is very strange, it shows either lack of understanding, or some twisted terminology.
It shows twisted terminology. I rewrote the main post to try to fix it.
I’d like to delete the whole post in shame, but I’m still confused as to whether we can be expected utility maximizers without being average utilitarianists.
I’ve thought about this a bit more, and I’m back to the intuition that you’re mixing up different concepts of “utility” somewhere, but I can’t make that notion any more precise. You seem to be suggesting that certain seemingly plausible preferences cannot be properly expressed as utility functions. Can you give a stripped-down, “single-player” example of this that doesn’t involve other people or selves?
Here’s a restatement:
We have a utility function u(outcome) that gives a utility for one possible outcome.
We have a utility function U(lottery) that gives a utility for a probability distribution over all possible outcomes.
The von Neumann-Morgenstern theorem indicates that the only reasonable form for U is to calculate the expected value of u(outcome) over all possible outcomes.
This means that your utility function U is indifferent with regard to whether the distribution of utility is equitable among your future selves. Giving one future self u=10 and another u=0 is equally as good as giving one u=5 and another u=5.
This is the same sort of ethical judgement that an average utilitarian makes when they say that, to calculate social good, we should calculate the average utility of the population.
Therefore, I think that the von Neumann-Morgenstern theorem does not prove, but provides very strong reasons for thinking, that average utilitarianism is correct.
And yet, average utilitarianism asserts that equity of utility, even among equals has no utility. This is shocking.
If you want a more equitable distribution of utility among future selves, then your utility function u(outcome) may be a different function than you thought it was; e.g. the log of the function you thought it was.
More generally, if u is the function that you thought was your utility function, and f is any monotonically increasing function on the reals with f″ < 0, then by Jensen’s inequality, an expected f″(u)-maximizer would prefer to distribute u-utility equitably among its future selves.
Exactly. (I didn’t realize the comments were continuing down here and made the essentially same point here after Phil amended the post.)
The interesting point that Phil raises is whether there’s any reason to have a particular risk preference with respect to u. I’m not sure that the analogy between being inequality averse amongst possible “me”s and and inequality averse amongst actual others gets much traction once we remember that probability is in the mind. But it’s an interesting question nonetheless.
Allais, in particular argued that any form of risk preference over u should be allowable, and Broome finds this view “very plausible”. All of which seems to make rational decision-making under uncertainty much more difficult, particularly as it’s far from obvious that we have intuitive access to these risk preferences. (I certainly don’t have intuitive access to mine.)
P.S. I assume you mean f(u)-maximizer rather than f″(u)-maximizer?
Yes, I did mean an f(u)-maximizer.
Yes—and then the f(u)-maximizer is not maximizing expected utility! Maximizing expected utility requires not wanting equitable distribution of utility among future selves.
Nope. You can have u(10 people alive) = −10 and u(only 1 person is alive)=100 or u(1 person is OK and another suffers)=100 and u(2 people are OK)=-10.
Not unless you mean something very different than I do by average utilitarianism.
I objected to drawing the analogy, and gave the examples that show where the analogy breaks. Utility over specific outcomes values the whole world, with all people in it, together. Alternative possibilities for the whole world figuring into the expected utility calculation are not at all the same as different people. People that the average utilitarianism talks about are not from the alternative worlds, and they do not each constitute the whole world, the whole outcome. This is a completely separate argument, having only surface similarity to the expected utility computation.
Maybe I’m missing the brackets between your conjunctions/disjunctions, but I’m not sure how you’re making a statement about Average Utilitarianism.
I’m with you so far.
What do you mean by “distribute utility to your future selves”? You can value certain circumstances involving future selves higher than others, but when you speak of “their utility” you’re talking about a completely different thing than the term u in your current calculation. u already completely accounts for how much they value their situation and how much you care whether or not they value it.
I don’t see how this at all makes the case for adopting average utilitarianism as a value framework, but I think I’m missing the connection you’re trying to draw.
I’d hate to see it go. I think you’ve raised a really interesting point, despite not communicating it clearly (not that I can probably even verbalize it yet). Once I got your drift it confused the hell out of me, in a good way.
Assuming I’m correct that it was basically unrelated, I think your previous talk of “happiness vs utility” might have primed a few folks to assume the worst here.
Phil, you’re making a claim that what others say about utility (i.e. that it’s good to maximize its expectation) is wrong. But it’s only on your idiosyncratic definition of utility that your argument has any traction.
You are free to use words any way you want (even if I personally find your usage frustrating at times). But you are not free to redefine others’ terms to generate an artificial problem that isn’t really there.
The injunction to “maximize expected utility” is entirely capable of incorporating your concerns. It can be “inequality-averse” if you want, simply by making it a concave function of experienced utility.
No. I’ve said this 3 times already, including in the very comment that you are replying to. The utility function is not defined across all possible outcomes. A utility function is defined over a single outcome; it evaluates a single outcome. It can discount inequalities within that outcome. It cannot discount across possible worlds. If it operated across all possible worlds, all you would say is “maximize utility”. The fact that you use the word “expected” means “average over all possible outcomes”. That is what “expected” means. It is a mathematical term whose meaning is already established.
You can safely ignore my previous reply, I think I finally see what you’re saying. Not sure what to make of it yet, but I was definitely misinterpreting you.
Repeating your definition of a utility function over and over again doesn’t oblige anybody else to use it. In particular, it doesn’t oblige all those people who have argued for expected utility maximization in the past to have adopted it before you tried to force it on them.
A von Neumann-Morgenstern utility function (which is what people are supposed to maximize the expectation of) is a representation of a set of consistent preferences over gambles. That is all it is. If your proposal results in a set of consistent preferences over gambles (I see no particular reason for it not to, but I could be wrong) then it corresponds to expected utility maximization for some utility function. If it doesn’t, then either it is inconsistent, or you have a beef with the axioms that runs deeper than an analogy to average utilitarianism.
“Expected” means expected value of utility function of possible outcomes, according to the probability distribution on the possible outcomes.
If you don’t prefer 10% chance of 101 utilons to 100% chance of 10, then you can rescale your utility function (in a non-affine manner). I bet you’re thinking of 101 as “barely more than 10 times as much” of something that faces diminishing returns. Such diminishing returns should already be accounted for in your utility function.
No. I’ve explained this in several of the other comments. That’s why I used the term “utility function”, to indicate that diminishing returns are already taken into account.
This doesn’t sound right to me. Assuming “world” means “world at time t”, a utility function at the very least has type (World → Utilons). It maps a single world to a single utility measure, but it’s still defined over all worlds, the same way that (+3) is defined over all integers. If it was only defined for a single world it wouldn’t really be much of a function, it’d be a constant.
We use expected utility due to uncertainty. If we had perfect information, we could maximize utility by searching over all action sequences, computing utility for each resulting world, and returning the sequence with the highest total utility.
I think this illustrates the problem with your definition. The utility you’re maximizing is not the same as the “utility 101 for one future you”. You first have to map future you’s utility to just plain utility for any of this to make sense.
I meant “the domain of a utility function is a single world.”
However, it turns out that the standard terminology includes both utility functions over a single world (“outcome”), and a big utility function over all possible worlds (“lottery”).
My question/observation is still the same as it was, but my misuse of the terminology has mangled this whole thread.
The reason why an inequitable distribution of money is problematic is that money has diminishing marginal utility; so if a millionaire gives $1000 to a poor person, the poor person gains more than the millionaire loses.
If your instincts are telling you that an inequitable distribution of utility is bad, are you sure you’re not falling into the “diminishing marginal utility of utility” error that people have been empirically shown to exhibit? (can’t find link now, sorry, I saw it here).
That’s why I said “utility” instead of “money”.
Er, I know, I’m contrasting money and utility. Could you expand a little more on what you’re trying to say about my point?
The term “utility” means that I’m taking diminishing marginal returns into account.
My instincts are confused on the point, but my impression is that most people find average utilitarianism reprehensible.
Perhaps it would help if you gave a specific example of an action that (a) follows from average utilitarianism as you understand it, and (b) you believe most people would find reprehensible?
The standard answer is killing a person with below-average well-being*, assuming no further consequences follow from this. This assumes dying has zero disutility, however.
See comments on For The People Who Are Still Alive for lots of related discussion.
*The term “experienced utility” seems to be producing a lot of confusion. Utility is a decision-theoretic construction only. Humans, as is, don’t have utility functions.
It also involves maximizing average instantaneous welfare, rather than the average of whole-life satisfaction.
Yes, I’m surprised that it’s average rather than total utility is being measured. All other things being equal, twice as many people is twice as good to me.
The standard answer is killing a person with below-average well-being*, assuming no further consequences follow from this. This assumes dying has zero disutility, however.
See comments on For The People Who Are Still Alive for lots of related discussion.
*I consider the term “experienced utility” harmful. Utility is a decision-theoretic abstraction, not an experience.
Dead people presumably count as zero utility. I was rather frightened before I saw that—if you only count living people, then you’d be willing to kill people for the crime of not being sufficiently happy or fulfilled.
This sentence doesn’t really mean much. A dead person doesn’t have preferences or utility (zero or otherwise) when dead any more than a rock does, a dead person had preferences and utility when alive. The death of a living person (who preferred to live) reduces average utility because the living person preferred to not die, and that preference is violated!
I support the right to euthanasia for people who truly prefer to be killed, e.g. because they suffer from terminal painful diseases. Do you oppose it?
Perhaps you should see it more clearly if you think of it as maximizing the average preference utility across the timeline, rather than the average utility at a single point in time.
But after the fact, they are not alive, so they do not impact the average utility across all living things, so you are increasing the average utility across all living things.
Here’s what I mean, roughly expressed. Two possible timelines.
(A)
2010 Alice loves her life (she wants to live with continued life being a preference satisfaction of 7 per year). Bob merely likes his life (he wants to live with continued life being a preference satisfaction of 6 per year).
2011 Alice and 2011 Bob both alive as before.
2012 Alice and 2012 Bob both alive as before.
2013 Alice and 2013 Bob both alive as before.
2010 Bob wants 2010 Bob to exist, 2011 Bob to exist, 2012 Bob to exist, 2013 Bob to exist.
2010 Alice wants 2010 Alice to exist, 2011 Alice to exist, 2012 Alice to exist, 2013 Alice to exist.
2010 average utility is therefore (4x7 + 4x6) / 2= 26 and that also remains the average for the whole timeline.
(B)
2010 Alice and Bob same as before.
2011 Alice is alive. Bob has just been killed.
2012 Alice alive as before.
2013 Alice alive as before.
2010 average utility is: 4x7 +6 = (4x7 + 1x6)/2 = 17
2011 average utility is: 4x7 = 28
2012 average utility is: 4x7 = 28
2013 average utility is: 4x7 = 28
So Bob’s death increased the average utility indicated in the preferences of a single year. But average utilty across the timeline is now (28 + 6 + 28 + 28 + 28) / 5 = 23.6
In short the average utility of the timeline as a whole is decreased by taking out Bob.
You are averaging based on the population at the start of the experiment. In essence, you are counting dead people in your average, like Eliezer’s offhanded comment implied he would. Also, you are summing over the population rather than averaging.
Correcting those discrepancies, we would see (ua ⇒ “utils average”; murder happening New Year’s Day 2011):
The murder was a clear advantage.
Now, let’s say we are using procreation instead of murder as the interesting behavior. Let’s say each act of procreation reduces the average utility by 1, and it starts at 100 at the beginning of the experiment, with an initial population of 10.
In the first year, we can decrease the average utility by 10 in order to add one human with 99 utility. When do we stop adding humans? Well, it’s clear that the average utility in this contrived example is equal to 110 minus the total population, and the total utility is equal to the average times the population size. If we have 60 people, that means our average utility is 50, with a total of 3,000 utils. Three times as good for everyone, except half as good for our original ten people.
We maximize the utility at a population of 55 in this example (and 55 average utility) -- but that’s because we can’t add new people very efficiently. If we had a very efficient way of adding more people, we’d end up with the average utility being just barely better than death, but we’d make up for it in volume. That’s what you are suggesting we do.
That also isn’t a universe I want to live in. Eliezer is suggesting we count dead people in our averages, nothing more. That’s sufficient to go from kill-almost-all-humans to something we can maybe live with. (Of course, if we counted their preferences, that would be a conservatizing force that we could never get rid of, which is similarly worrying, albeit not as much so. In the worst case, we could use an expanding immortal population to counter it. Still-living people can change their preferences.)
I consider every moment of living experience as of equal weight. You may call that “counting dead people” if you want, but that’s only because when considering the entire timeline I consider every living moment—given a single timeline, there’s no living people vs dead people, there’s just people living in different times. If you calculate the global population it doesn’t matter what country you live in—if you calculate the utility of a fixed timeline, it doesn’t matter what time you live in.
But the main thing I’m not sure you get is that I believe preferences are valid also when concerning the future, not just when concerning the present.
If 2014 Carl wants the state of the world to be X in 2024, that’s still a preference to be counted, even if Carl ends up dead in the meantime. That Carl severely does NOT want to be dead in 2024, means that there’s a heavy disutility penalty for the 2014 function of his utility if he ends up nonetheless dead in 2024.
If e.g. someone wants to be buried at sea because he loves the sea, I consider it good that we bury them at sea.
But if someone wants to be buried at sea only because he believes such a ritual is necessary for his soul to be resurrected by God Poseidon, his preference is dependent on false beliefs—it doesn’t represent true terminal values; and that’s the ones I’m concerned about.
If conservatism is e.g. motivated by either wrong epistemic beliefs, or by fear, rather than true different terminal values, it should likewise not modify our actions, if we’re acting from an epistemically superior position (we know what they didn’t).
That’s an ingenious fix, but when I think about it I’m not sure it works. The problem is that although you are calculating the utility integrated over the timeline, the values that you are integrating are still based on a particular moment. In other words, calculating the utility of the 2014-2024 timeline by 2014 preferences might not produce the same result as calculating the utility of the 2014-2024 timeline by 2024 preferences. Worse yet, if you’re comparing two timelines and the two timelines have different 2024s in them, and you try to compare them by 2024 preferences, which timeline’s 2024 preferences do you use?
For instance, consider timeline A: Carl is alive in 2014 and is killed soon afterwards, but two new people are born who are alive in 2024. timeline B: Carl is alive in 2014 and in 2024, but the two people from A never existed.
If you compare the timelines by Carl’s 2014 preferences or Carl’s timeline B 2024 preferences, timeline B is better, because timeline B has a lot of utility integrated over Carl’s life. If you compare the timelines by the other people’s timeline A 2024 preferences, timeline A is better.
It’s tempting to try to fix this argument by saying that rather than using preferences at a particular moment, you will use preferences integrated over the timeline, but if you do that in the obvious way (by weighting the preferences according to the person-hours spent with that preference), then killing someone early reduces the contribution of their preference to the integrated utility, causing a problem similar to the original one.
I think you’re arguing against my argument against a position you don’t hold, but which I called by a term that sounds to you like your position.
Assuming you have a function that yields the utility that one person has at one particular second, what do you want to optimize for?
And maybe I should wait until I’m less than 102 degrees Fahrenheit to continue this discussion.
I believe that what dhasenan was getting at is that without the assumption that a dead person has 0 utility, you would be willing to kill people who are happy (positive utility), but just not as happy as they could be. I’m not sure how exactly this would go mathematically, but the point is that killing a +utility person being a reduction in utility is a vital axiom
It’s not that they could be happier. Rather, if the average happiness is greater than my happiness, the average happiness in the population will be increased if I die (assuming the other effects of a person dying are minimal or sufficiently mitigated).
I don’t know if we need have it as an axiom rather than this being a natural consequence of happy people preferring not to be killed, and of us likewise preferring not to kill them, and of pretty much everyone preferring their continued lives to their deaths… The good of preference utilitarianism is that it takes all these preferences as an input.
If preference average utilitarianism nonetheless leads to such an abominable conclusion, I’ll choose to abandon preference average utilitarianism, considering it a failed/misguided attempt at describing my sense of morality—but I’m not certain it needs lead to such a conclusion at all.
If I agreed, I’d be extremely curious as to what the average utility for all people across the multiverse actually is. (Is it dominated by people with extremely short lifespans, because they use so little computing power in a 4D sense?)
On average? 67 utilons.
If you want to maximize a value, please don’t compute it unconditionally, instead compute its dependence on your actions.
This seems like a useful request to make in a different context. It doesn’t appear relevant to the grandparent.
Why? Eliezer said he wanted to maximize the average utility of all people, then said that average utility was 67. Now he faces the difficult task of maximizing 67. Or maybe maximizing the utility of people who share a planet with him, at the expense of other people in existence, so the average stays 67. Am I missing something? :-)
Excuse me, what’s “average utility”? How do you compare utils of different people? Don’t say you’re doing it through the lens of your own utility function—this is begging the question.
Coherent Extrapolated Volition tries to resolve conflicting individual utilities—see “Fred wants to kill Steve”.
At this point, it looks like resolving conflicts should just be carried out as cooperation of individual preferences.
They’re easy to help if their utility functions included terms that outlived them, e.g. “world peace forever”. But it still feels somehow wrong to include them in the calculation, because this will necessarily be at the expense of living and future people.
In my non-professional capacity, when I try to help others, I’m doing so to optimize my utility function over them: I want them to be happy, and only living people can fulfill this aspect of my utility function. It’s in this sense that I say “we should” meaning “altruists should do the analogue for their own utility functions”.