Only if they can see through the illusion in the first place
Well, yes, but the question is what humanity wants. Humanity does not want that to be done to it. It’s not what CEV would output, and if you’re in the position to implement a wrapper-mind with some utility function of your choice, putting that in it would be a violation of human preferences on a cosmological scale. That’s very much not leaving your fingerprints off the future.
consider an angry teenager who break up with their partner and firmly says they want to immediately die
That’s probably not what they’d actually want on reflection (i. e., if you ran the value conflict resolution and value extrapolation steps), just what they want in the moment.
“Humanity” is not a single agent. It can have preferences only insofar as the individual humans share similar preferences. If you are a happiness maximizer, for every individual human you look at, you’ll probably find that their happiness would be maximized by wireheading (because most whimsical desires like becoming the king of the world are not feasible to satisfy otherwise).
I’m not even that sure that CEV would avoid this problem. In which way being enclosed in a perfect world is not the best thing for you? Because it would be fake? But how do you know that’s fake? Imagine that an AGI offers to teleport you on another planet, which is perfectly suitable for you; you’ll land there and thrive forever. Now imagine that instead of actually teleporting you to another planet, it just let you to believe the same; you’ll (mentally) land there and thrive forever anyway. I mean, your brain is experiencing the very same thing! It’s not obvious that the second option is worse than the first one, unless you have an hardcoded limit like “Thou Shalt Not Wirehead People”.
Well, yes, but the question is what humanity wants. Humanity does not want that to be done to it. It’s not what CEV would output, and if you’re in the position to implement a wrapper-mind with some utility function of your choice, putting that in it would be a violation of human preferences on a cosmological scale. That’s very much not leaving your fingerprints off the future.
That’s probably not what they’d actually want on reflection (i. e., if you ran the value conflict resolution and value extrapolation steps), just what they want in the moment.
“Humanity” is not a single agent. It can have preferences only insofar as the individual humans share similar preferences. If you are a happiness maximizer, for every individual human you look at, you’ll probably find that their happiness would be maximized by wireheading (because most whimsical desires like becoming the king of the world are not feasible to satisfy otherwise).
I’m not even that sure that CEV would avoid this problem. In which way being enclosed in a perfect world is not the best thing for you? Because it would be fake? But how do you know that’s fake? Imagine that an AGI offers to teleport you on another planet, which is perfectly suitable for you; you’ll land there and thrive forever. Now imagine that instead of actually teleporting you to another planet, it just let you to believe the same; you’ll (mentally) land there and thrive forever anyway. I mean, your brain is experiencing the very same thing! It’s not obvious that the second option is worse than the first one, unless you have an hardcoded limit like “Thou Shalt Not Wirehead People”.