What decision is made when multiple choices all leave the variables within tolerance?
Whatever occurs to us first. ;-)
What decision is made when none of the available choices leave the variables within tolerance?
We waffle, or try to avoid making the decision in the first place. ;-) (See, e.g., typical people’s reactions to “trolley problems”, or other no-win scenarios.)
It is simply “that thing that gets maximized in any particular person’s decision making”. Perhaps you think that humans do not maximize utility because you have a preferred definition of utility that is different from this one
What I’m saying is that the above construction leads to error if you assume that “utility” is a function of the state of the world outside the human, rather than a function of the difference between the human’s perceptions of the outside world, and the human’s internal reference values or tolerance ranges for those perceptions.
Maximizing a utility function over the state of the external world inherently tends to create results that would be considered undesirable by most humans. (See, for example, the various tortured insanities that come about when you try to maximize such a conception of “utility” over a population of humans.)
It’s important to understand that the representation you use to compute something is not value-neutral. Roman numerals, for example, make division much more complicated than Arabic ones.
So, I’m not saying that you can’t create some sort of “utility” function to represent human values. We have no reason to assume that human values aren’t Turing-computable, and if they’re Turing-computable, we should be able to use whatever stupidly complex representation we want to compute them.
However, to use world-state-utility as your basis for computation is just plain silly, like using Roman numerals for long division. Your own intuition will make it harder for you to see the Friendliness-failures that are sitting right under your nose, because utility maximization is utterly foreign to normal human cognitive processes. (Externality-maximizing processes in human behavior are generally the result of pathology, rather than normal brain function.)
But the maximizing theory has been under scrutiny for 150 years, and under strong scrutiny for the past 50.
Eliezer hasn’t been alive that long, has he? ;-)
Seriously, though, external-utility-maximizing thinking is the very essence of Unfriendly AI, and the history of discussions of world-state-based utility is that models based on it lead to counterintuitive results unless you torture the utility function hard enough, and/or carefully avoid the sort of creative thinking that an unfettered superintelligence might come up with.
Whatever occurs to us first. ;-)
We waffle, or try to avoid making the decision in the first place. ;-) (See, e.g., typical people’s reactions to “trolley problems”, or other no-win scenarios.)
What I’m saying is that the above construction leads to error if you assume that “utility” is a function of the state of the world outside the human, rather than a function of the difference between the human’s perceptions of the outside world, and the human’s internal reference values or tolerance ranges for those perceptions.
Maximizing a utility function over the state of the external world inherently tends to create results that would be considered undesirable by most humans. (See, for example, the various tortured insanities that come about when you try to maximize such a conception of “utility” over a population of humans.)
It’s important to understand that the representation you use to compute something is not value-neutral. Roman numerals, for example, make division much more complicated than Arabic ones.
So, I’m not saying that you can’t create some sort of “utility” function to represent human values. We have no reason to assume that human values aren’t Turing-computable, and if they’re Turing-computable, we should be able to use whatever stupidly complex representation we want to compute them.
However, to use world-state-utility as your basis for computation is just plain silly, like using Roman numerals for long division. Your own intuition will make it harder for you to see the Friendliness-failures that are sitting right under your nose, because utility maximization is utterly foreign to normal human cognitive processes. (Externality-maximizing processes in human behavior are generally the result of pathology, rather than normal brain function.)
Eliezer hasn’t been alive that long, has he? ;-)
Seriously, though, external-utility-maximizing thinking is the very essence of Unfriendly AI, and the history of discussions of world-state-based utility is that models based on it lead to counterintuitive results unless you torture the utility function hard enough, and/or carefully avoid the sort of creative thinking that an unfettered superintelligence might come up with.