I am interested in answering questions of “what to want.” Not only is it important for individual decision-making, but there are also many interesting ethical questions. If a person’s utility function can be changed through experience, is it ethical to steer it in a direction that would benefit you? Take the example of religion: suppose you could convince an individual to convert to a religion, and then further convince them to actively reject new information that would endanger their faith. Is this ethical? (My opinion is that it depends on your own motivations. If you actually believed in the religion, then you might be convinced that you are benefiting others by converting them. If you did not actually believe in the religion, then you are being manipulative.)
I have to admit that, for myself, I remain unconvinced that there is an objective truth to be had regarding “what should I want”. Partly because I’m unconvinced that “I” is a coherent unitary thing at any given timepoint, let alone over time. And partly because I don’t see how to distinguish “preferences” from “tendencies” without resorting to unmeasurable guesses about qualia and consciousness.
Thanks for the references.
I am interested in answering questions of “what to want.” Not only is it important for individual decision-making, but there are also many interesting ethical questions. If a person’s utility function can be changed through experience, is it ethical to steer it in a direction that would benefit you? Take the example of religion: suppose you could convince an individual to convert to a religion, and then further convince them to actively reject new information that would endanger their faith. Is this ethical? (My opinion is that it depends on your own motivations. If you actually believed in the religion, then you might be convinced that you are benefiting others by converting them. If you did not actually believe in the religion, then you are being manipulative.)
Cool. The (Metaethics Sequence)[http://wiki.lesswrong.com/wiki/Metaethics_sequence] is useful for some of those things.
I have to admit that, for myself, I remain unconvinced that there is an objective truth to be had regarding “what should I want”. Partly because I’m unconvinced that “I” is a coherent unitary thing at any given timepoint, let alone over time. And partly because I don’t see how to distinguish “preferences” from “tendencies” without resorting to unmeasurable guesses about qualia and consciousness.