I think what the “true” (status-quo) ending proves is that the Super-Happies did not accurately model humanity’s utility function at all. If they had, they would have proposed a deal where humanity gets rid of most of its pain, but still keeps some, especially those “grim” things that humans actually like (somewhat counter-intuitively). (And perhaps the Babyeaters’ thing would then be understood as one of these “grim” things by humans, as it clearly is for the Babyeaters themselves It’s not clear if the Superhappies would be willing to acquire this value, though). This is a deal that humans would indeed accept, since it agrees with their values. I think the true moral of this story is that getting human wants right for something like CEV is a hard problem, and making even small mistakes can have big consequences.
I think what the “true” (status-quo) ending proves is that the Super-Happies did not accurately model humanity’s utility function at all. If they had, they would have proposed a deal where humanity gets rid of most of its pain, but still keeps some, especially those “grim” things that humans actually like (somewhat counter-intuitively). (And perhaps the Babyeaters’ thing would then be understood as one of these “grim” things by humans, as it clearly is for the Babyeaters themselves It’s not clear if the Superhappies would be willing to acquire this value, though). This is a deal that humans would indeed accept, since it agrees with their values. I think the true moral of this story is that getting human wants right for something like CEV is a hard problem, and making even small mistakes can have big consequences.