I think that this confusion results from failing to distinguish between your individual utility function and the “effective social utility function” (the result of cooperative bargaining between all individuals in a society). The individual utility function is bounded on a scale which is roughly comparable to Dunbar’s number[1]. The effective social utility function is bounded on a scale comparable to the current size of humanity. When you conflate them, the current size of humanity seems like a strangely arbitrary parameter so you’re tempted to decide the utility function is unbounded.
The reason why distinguishing between those two is so hard, is because there are strong social incentives to conflate them, incentives which our instincts are honed to pick up on. Pretending to unconditionally follow social norms is a great way to seem trustworthy. When you combine it with an analytic mindset that’s inclined to reasoning with explicit utility functions, this self-deception takes the form of modeling your intrinsic preferences by utilitarianism.
Another complication is, larger universes tend to be more diverse and hence more interesting. But this also saturates somewhere (having e.g.10100 books to choose from is not noticeably better from having 1050 books to choose from).
I think that this confusion results from failing to distinguish between your individual utility function and the “effective social utility function” (the result of cooperative bargaining between all individuals in a society). The individual utility function is bounded on a scale which is roughly comparable to Dunbar’s number[1]. The effective social utility function is bounded on a scale comparable to the current size of humanity. When you conflate them, the current size of humanity seems like a strangely arbitrary parameter so you’re tempted to decide the utility function is unbounded.
The reason why distinguishing between those two is so hard, is because there are strong social incentives to conflate them, incentives which our instincts are honed to pick up on. Pretending to unconditionally follow social norms is a great way to seem trustworthy. When you combine it with an analytic mindset that’s inclined to reasoning with explicit utility functions, this self-deception takes the form of modeling your intrinsic preferences by utilitarianism.
Another complication is, larger universes tend to be more diverse and hence more interesting. But this also saturates somewhere (having e.g.10100 books to choose from is not noticeably better from having 1050 books to choose from).
It seems plausible to me both for explaining how people behave in practice and in terms of evolutionary psychology.