I figured out what the problem is. Axiom 4 (Independence) implies average utilitarianism is correct.
Suppose you have two apple pies, and two friends, Betty and Veronica. Let B denote the number of pies you give to Betty, and V the number you give to Veronica. Let v(n) denote the outcome that Veronica gets n apple pies, and similarly define b(n). Let u_v(S) denote Veronica’s utility in situation S, and u_b(S) denote Betty’s utility.
Betty likes apple pies, but Veronica loves them, so much so that u_v(v(2), b(0)) > u_b(b(1), v(1)) + u_v(b(1), v(1)).
We want to know whether average utilitarianism is correct to know whether to give Veronica both pies.
Independence, the fourth axiom of the von Neumann-Morgenstern theorem, implies that if the outcome L is preferable to outcome M, then one outcome of L and one outcome of N is preferable to one outcome of M and one outcome of N.
Let L represent giving one pie to Veronica and M represent giving one pastry to Betty. Now let’s be sneaky and let N also represent giving one pastry to Veronica. The fourth axiom says that L + N—giving two pies to Veronica—is preferable to L + M—giving one to Veronica and one to Betty. We have to assume that to use the theorem.
But that’s the question we wanted to ask—whether our utility function U should prefer the solution that gives two pies to Veronica, or one to Betty and one to Veronica! Assuming the fourth axiom builds average utilitarianism into the von Neumann-Morgenstern theorem.
I figured out what the problem is. Axiom 4 (Independence) implies average utilitarianism is correct.
Suppose you have two apple pies, and two friends, Betty and Veronica. Let B denote the number of pies you give to Betty, and V the number you give to Veronica. Let v(n) denote the outcome that Veronica gets n apple pies, and similarly define b(n). Let u_v(S) denote Veronica’s utility in situation S, and u_b(S) denote Betty’s utility.
Betty likes apple pies, but Veronica loves them, so much so that u_v(v(2), b(0)) > u_b(b(1), v(1)) + u_v(b(1), v(1)). We want to know whether average utilitarianism is correct to know whether to give Veronica both pies.
Independence, the fourth axiom of the von Neumann-Morgenstern theorem, implies that if the outcome L is preferable to outcome M, then one outcome of L and one outcome of N is preferable to one outcome of M and one outcome of N.
Let L represent giving one pie to Veronica and M represent giving one pastry to Betty. Now let’s be sneaky and let N also represent giving one pastry to Veronica. The fourth axiom says that L + N—giving two pies to Veronica—is preferable to L + M—giving one to Veronica and one to Betty. We have to assume that to use the theorem.
But that’s the question we wanted to ask—whether our utility function U should prefer the solution that gives two pies to Veronica, or one to Betty and one to Veronica! Assuming the fourth axiom builds average utilitarianism into the von Neumann-Morgenstern theorem.
Argh; never mind. This is what Wei_Dai already said below.