I have a question for Eliezer. I went back and reread your sequence on metaethics, and the amount of confusion in the comments struck me, so now I want to make sure that I understood you correctly. After rereading, my interpretation didn’t change, but I’m still unsure. So, does this summarize your position accurately:
A simple mind has a bunch of terminal values (or maybe one) summarized in a utility function. Morality for it, or rather not morality, but the thing this mind has which is analogous to morality in humans (depending on how you define “morality”) is summed up in this utility function. This is the only source of shouldness for that simple mind.
For humans, the situation is more complex. We have preferences which are like a utility function, but aren’t because we aren’t expected utility maximizers. Moreover, these preferences change depending on a number of factors. But this isn’t the source of shouldness we are looking for. Buried deep in the human mind is a legitimate utility function, or at least something like one, which summarizes that human’s terminal values, thus providing that source of shouldness. This utility function is very hard to discover due to the psychology of humans, but it exists. The preference set of any given human has is an approximation of that human’s utility function (though not necessarily a good one) subject, of course, to the many biases humans are fraught with.
The final essential point is that, due to the psychological unity of mankind, the utility functions of each person are likely to be very similar, if not the same, so when we call something “right” or “moral” we are referring to (nearly) the same thing.
But this isn’t the source of shouldness we are looking for. Buried deep in the human mind is a legitimate utility function, or at least something like one, which summarizes that human’s terminal values
No. It’s more that if you extrapolate out the preferences we already have, asking what we would prefer if we had time for our chaotic preferences to resolve themselves, then you end up with a superior sort of shouldness to which our present preferences might well defer. Sort of like if you knew that your future self would be a vegetarian, you might regard your present consumption of meat as an error. But it’s not hidden away as something that already exists. It’s something that could be computed from us, but which we don’t explicitly represent.
Hence “deep in the mind”, not brain: defined in a model, not explicitly represented. Although there is more preference-defining stuff outside the mind (or rather outside the brain...).
To be honest, I wasn’t thinking of the the distinction between mind and brain when I wrote that, so Eliezer’s correction is on target. I was visualizing the utility function as something that exists and must be discovered.
I’m not sure what do you see in the distinction between simple preference and complex preference. No matter how simple an imperfect agent is, you face a problem of going from imperfect decision-making to ideal preference order.
I’m not sure what do you see in the distinction between simple preference and complex preference. No matter how simple an imperfect agent is, you face a problem of going from imperfect decision-making to ideal preference order.
I don’t mean simple or complicated preferences. I mean a simple mind (perhaps simple was a bad choice of terminology). My “simple mind” is a mind that perfectly knows it’s utility function (and has a well-defined utility function to begin with). It’s just an abstraction to better understand where shouldness comes from.
There is a large complication in that we call something “moral” when we want other people to do it. So there are probably things that we call “moral” that are actually “sins” according to our internal utility functions.
I have a question for Eliezer. I went back and reread your sequence on metaethics, and the amount of confusion in the comments struck me, so now I want to make sure that I understood you correctly. After rereading, my interpretation didn’t change, but I’m still unsure. So, does this summarize your position accurately:
A simple mind has a bunch of terminal values (or maybe one) summarized in a utility function. Morality for it, or rather not morality, but the thing this mind has which is analogous to morality in humans (depending on how you define “morality”) is summed up in this utility function. This is the only source of shouldness for that simple mind.
For humans, the situation is more complex. We have preferences which are like a utility function, but aren’t because we aren’t expected utility maximizers. Moreover, these preferences change depending on a number of factors. But this isn’t the source of shouldness we are looking for. Buried deep in the human mind is a legitimate utility function, or at least something like one, which summarizes that human’s terminal values, thus providing that source of shouldness. This utility function is very hard to discover due to the psychology of humans, but it exists. The preference set of any given human has is an approximation of that human’s utility function (though not necessarily a good one) subject, of course, to the many biases humans are fraught with.
The final essential point is that, due to the psychological unity of mankind, the utility functions of each person are likely to be very similar, if not the same, so when we call something “right” or “moral” we are referring to (nearly) the same thing.
Does that sound right?
No. It’s more that if you extrapolate out the preferences we already have, asking what we would prefer if we had time for our chaotic preferences to resolve themselves, then you end up with a superior sort of shouldness to which our present preferences might well defer. Sort of like if you knew that your future self would be a vegetarian, you might regard your present consumption of meat as an error. But it’s not hidden away as something that already exists. It’s something that could be computed from us, but which we don’t explicitly represent.
Hence “deep in the mind”, not brain: defined in a model, not explicitly represented. Although there is more preference-defining stuff outside the mind (or rather outside the brain...).
To be honest, I wasn’t thinking of the the distinction between mind and brain when I wrote that, so Eliezer’s correction is on target. I was visualizing the utility function as something that exists and must be discovered.
Sounds about right, except that I wouldn’t call this anything close to a summary of the whole position. Also, compare the status of morality with that of probability (e.g. Probability is Subjectively Objective, Can Counterfactuals Be True?, Math is Subjunctively Objective).
I’m not sure what do you see in the distinction between simple preference and complex preference. No matter how simple an imperfect agent is, you face a problem of going from imperfect decision-making to ideal preference order.
I don’t mean simple or complicated preferences. I mean a simple mind (perhaps simple was a bad choice of terminology). My “simple mind” is a mind that perfectly knows it’s utility function (and has a well-defined utility function to begin with). It’s just an abstraction to better understand where shouldness comes from.
The (effective) utility functions are different enough to produce fighting and wars.
The problem is that the utility functions refer to “me”—and that’s different in every single case.
There is a large complication in that we call something “moral” when we want other people to do it. So there are probably things that we call “moral” that are actually “sins” according to our internal utility functions.