I think the issue may be that the “egoistic” utility functions are incoherent in this context, because you’re actually trying to compare the utility functions of two different agents as if they were one.
Let’s say, for example, that X is a paperclip maximiser who gets either 10 paperclips or 9 paperclips, and each X* is a human who either saves 2 million lives or 1 million lives.
If you don’t know whether you’re X or X*, how can you compare 10 paperclips to 2 million lives?
I think the issue may be that the “egoistic” utility functions are incoherent in this context, because you’re actually trying to compare the utility functions of two different agents as if they were one.
Let’s say, for example, that X is a paperclip maximiser who gets either 10 paperclips or 9 paperclips, and each X* is a human who either saves 2 million lives or 1 million lives.
If you don’t know whether you’re X or X*, how can you compare 10 paperclips to 2 million lives?