My main objection for the simplified utility functions is that they are presented as depending only upon the current external state of the world in some vaguely linear and stable way. Every adjective in there corresponds to discarding a lot of useful information about preferences that people actually have.
People often have strong preferences about potential pasts, presents, and future as well as the actual present. This includes not just things like how things are, but also about how things could have gone. I would be very dissatisfied if some judges had flipped coins to render a verdict, even if by chance every verdict was correct and the usual process would have delivered some incorrect verdicts.
People have rather strong preferences about their own internal states, not just about the external universe. For example, intransitive preferences are usually supposed to be pumpable, but this neglects the preference people have for not feeling ripped off and similar internal states. This also ties into the previous example where I would feel a justified loss of confidence in the judicial system which is unpleasant in itself, not just in its likelihood of affecting my life or those I care about in the future.
People have path-dependent preferences, not just preferences for some outcome state or other. For example, they may prefer a hypothetical universe in which some people were never born to one in which some people were born, lived, and then were murdered in secret. The final outcomes may be essentially identical, but can be very different in preference orderings.
People often have very strongly nonlinear preferences. Not just smoothly nonlinear, but outright discontinuous. They can also change over time for better or worse reasons, or for none at all.
Decision theories based on eliminating all these real phenomena seem very much less than useful.
My main objection for the simplified utility functions is that they are presented as depending only upon the current external state of the world in some vaguely linear and stable way. Every adjective in there corresponds to discarding a lot of useful information about preferences that people actually have.
The main argument I’ve heard for this kind of simplification is that your altruistic, morality-type preferences ought to be about the state of the external world because their subject is the wellbeing of other people, and the external world is where other people live. The linearity part is sort of an extension of the principle of treating people equally. I might be steelmanning it a little, a lot of times the argument is less that and more that having preferences that are in any way weird or complex is “arbitrary.” I think this is based on the mistaken notion that “arbitrary” is a synonym for “picky” or “complicated.”
I find this argument unpersuasive because altruism is also about respecting the preferences of others, and the preferences of others are, as you point out, extremely complicated and about all sorts of things other than the current state of the external world. I am also not sure that having nonlinear altruistic preferences is the same thing as not valuing people equally. And I think that our preferences about the welfare of others are often some of the most path-dependent preferences that we have.
EDIT: I have sense found this post, which discusses some similar arguments and refutes them more coherently than I do.
Second EDIT: I still find myself haunted by the “scary situation” I linked to and find myself wishing there was a way to tweak a utility function a little to avoid it, or at least get a better “exchange rate” than “double tiny good thing and more-than doubling horrible thing while keeping probability the same.” I suppose there must be a way since the article I linked to said it would not work on all bounded utility functions.
My main objection for the simplified utility functions is that they are presented as depending only upon the current external state of the world in some vaguely linear and stable way. Every adjective in there corresponds to discarding a lot of useful information about preferences that people actually have.
People often have strong preferences about potential pasts, presents, and future as well as the actual present. This includes not just things like how things are, but also about how things could have gone. I would be very dissatisfied if some judges had flipped coins to render a verdict, even if by chance every verdict was correct and the usual process would have delivered some incorrect verdicts.
People have rather strong preferences about their own internal states, not just about the external universe. For example, intransitive preferences are usually supposed to be pumpable, but this neglects the preference people have for not feeling ripped off and similar internal states. This also ties into the previous example where I would feel a justified loss of confidence in the judicial system which is unpleasant in itself, not just in its likelihood of affecting my life or those I care about in the future.
People have path-dependent preferences, not just preferences for some outcome state or other. For example, they may prefer a hypothetical universe in which some people were never born to one in which some people were born, lived, and then were murdered in secret. The final outcomes may be essentially identical, but can be very different in preference orderings.
People often have very strongly nonlinear preferences. Not just smoothly nonlinear, but outright discontinuous. They can also change over time for better or worse reasons, or for none at all.
Decision theories based on eliminating all these real phenomena seem very much less than useful.
The main argument I’ve heard for this kind of simplification is that your altruistic, morality-type preferences ought to be about the state of the external world because their subject is the wellbeing of other people, and the external world is where other people live. The linearity part is sort of an extension of the principle of treating people equally. I might be steelmanning it a little, a lot of times the argument is less that and more that having preferences that are in any way weird or complex is “arbitrary.” I think this is based on the mistaken notion that “arbitrary” is a synonym for “picky” or “complicated.”
I find this argument unpersuasive because altruism is also about respecting the preferences of others, and the preferences of others are, as you point out, extremely complicated and about all sorts of things other than the current state of the external world. I am also not sure that having nonlinear altruistic preferences is the same thing as not valuing people equally. And I think that our preferences about the welfare of others are often some of the most path-dependent preferences that we have.
EDIT: I have sense found this post, which discusses some similar arguments and refutes them more coherently than I do.
Second EDIT: I still find myself haunted by the “scary situation” I linked to and find myself wishing there was a way to tweak a utility function a little to avoid it, or at least get a better “exchange rate” than “double tiny good thing and more-than doubling horrible thing while keeping probability the same.” I suppose there must be a way since the article I linked to said it would not work on all bounded utility functions.