That seems to be a reasonable interpretation, but if we do interpret “the utility function isn’t up for grabs” that way, as a factual claim that each person has a utility function that can be discovered but not changed by moral arguments and reasoning, then I think it’s far from clear that the claim is true.
There could be other interpretations that may or may not be more plausible, and I’m curious what Eliezer’s own intended meaning is, as well as what pengvado meant by it.
There is a sense in which anything that makes choices does have a utility function—the utility function revealed by their choices. In this sense, for example, that akrasia doesn’t exist. People prefer to procrastinate, as revealed by their choice to procrastinate.
People frequently slip back and forth between this sense of “utility function” (a rather strange description of their behavior, whatever that is) and the utilitarian philosophers’ notions of “utility”, which have something to do with happiness/pleasure/fun. To the extent that people pursue happiness, pleasure, and fun, the two senses overlap. However, in my experience, people frequently make themselves miserable or make choices according to lawful rules (of morality, say) - without internal experiences of pleasure in following those rules.
And it’s worse than just akrasia. If you have incoherent preferences and someone money-pumps you, then the revealed utility function is “likes running around in circles”, i.e. it isn’t even about the choices you thought you were deciding between.
That seems to be a reasonable interpretation, but if we do interpret “the utility function isn’t up for grabs” that way, as a factual claim that each person has a utility function that can be discovered but not changed by moral arguments and reasoning, then I think it’s far from clear that the claim is true.
There could be other interpretations that may or may not be more plausible, and I’m curious what Eliezer’s own intended meaning is, as well as what pengvado meant by it.
There is a sense in which anything that makes choices does have a utility function—the utility function revealed by their choices. In this sense, for example, that akrasia doesn’t exist. People prefer to procrastinate, as revealed by their choice to procrastinate.
People frequently slip back and forth between this sense of “utility function” (a rather strange description of their behavior, whatever that is) and the utilitarian philosophers’ notions of “utility”, which have something to do with happiness/pleasure/fun. To the extent that people pursue happiness, pleasure, and fun, the two senses overlap. However, in my experience, people frequently make themselves miserable or make choices according to lawful rules (of morality, say) - without internal experiences of pleasure in following those rules.
And it’s worse than just akrasia. If you have incoherent preferences and someone money-pumps you, then the revealed utility function is “likes running around in circles”, i.e. it isn’t even about the choices you thought you were deciding between.
Yup.
Speaking as if “everyone” has a utility function is common around here, but it makes my teeth hurt.