For example, you might want you and everyone else to both be happy, and happiness of one without the other would be much less valuable.
Now you’ve got me curious. I don’t see what selections of values representative of the agent they’re trying to model could possibly desire non-Pareto-optimal scenarios. The given example (quoted), for one, is something I’d represent like this:
Let x = my happiness, y = happiness of everyone else
To model the fact that each is worthless without the other, let:
v1 = min(x, 10y) v2 = min(y, 10x)
Choice A: Gain 10 x, 0 y Choice B: Gain 0 x, 10 y Choice C: Gain 2 x, 2 y
It seems very obvious that the sole Pareto-optimal choice is the only desirable policy. Utility is four for choice C, and zero for A and B.
This may reduce to exactly what AlexMennen said, too, I guess. I have never encountered any intuition or decision problem that couldn’t at-least-in-principle resolve to a utility function with perfect modeling accuracy given enough time and computational resources.
I do think that everything should reduce to a single utility function. That said, this utility function is not necessarily a convex combination of separate values, such as “my happiness”, “everyone else’s happiness”, etc. It could contain more complex values such as your v1 and v2, which depend on both x and y.
In your example, let’s add a choice D: 50% of the time it’s A, 50% of the time it’s B. In terms of individual happiness, this is Pareto superior to C. It is Pareto inferior for v1 and v2, though.
EDIT: For an example of what I’m criticizing: Nisan claims that this theorem presents a difficulty for avoiding the repugnant conclusion if your desiderata are total and average happiness. If v1 = total happiness and v2 = average happiness, and Pareto optimality is desirable, then it follows that utility is a*v1 + b*v2. From this utility function, some degenerate behavior (blissful solipsist or repugnant conclusion) follows. However, there is nothing that says that Pareto optimality in v1 and v2 is desirable. You might pick a non-linear utility function of total and average happiness, for example atan(average happiness) + atan(total happiness). Such a utility function will sometimes pick policies that are Pareto inferior with respect to v1 and v2.
Now you’ve got me curious. I don’t see what selections of values representative of the agent they’re trying to model could possibly desire non-Pareto-optimal scenarios. The given example (quoted), for one, is something I’d represent like this:
Let x = my happiness, y = happiness of everyone else
To model the fact that each is worthless without the other, let:
v1 = min(x, 10y)
v2 = min(y, 10x)
Choice A: Gain 10 x, 0 y
Choice B: Gain 0 x, 10 y
Choice C: Gain 2 x, 2 y
It seems very obvious that the sole Pareto-optimal choice is the only desirable policy. Utility is four for choice C, and zero for A and B.
This may reduce to exactly what AlexMennen said, too, I guess. I have never encountered any intuition or decision problem that couldn’t at-least-in-principle resolve to a utility function with perfect modeling accuracy given enough time and computational resources.
I do think that everything should reduce to a single utility function. That said, this utility function is not necessarily a convex combination of separate values, such as “my happiness”, “everyone else’s happiness”, etc. It could contain more complex values such as your v1 and v2, which depend on both x and y.
In your example, let’s add a choice D: 50% of the time it’s A, 50% of the time it’s B. In terms of individual happiness, this is Pareto superior to C. It is Pareto inferior for v1 and v2, though.
EDIT: For an example of what I’m criticizing: Nisan claims that this theorem presents a difficulty for avoiding the repugnant conclusion if your desiderata are total and average happiness. If v1 = total happiness and v2 = average happiness, and Pareto optimality is desirable, then it follows that utility is a*v1 + b*v2. From this utility function, some degenerate behavior (blissful solipsist or repugnant conclusion) follows. However, there is nothing that says that Pareto optimality in v1 and v2 is desirable. You might pick a non-linear utility function of total and average happiness, for example atan(average happiness) + atan(total happiness). Such a utility function will sometimes pick policies that are Pareto inferior with respect to v1 and v2.