Being embedded in a fake reality and fooled into believing it’s true would be against many people’s preferences.
Only if they can see through the illusion in the first place. Suppose that the happiness-maximizing AGI could throw you into a perfect fake reality without you even noticing. One day you wake up, and from that point on everything goes perfectly well for you in some very plausible manner. After a while, your happiness has reached immensely high levels. “This poor schmuck doesn’t want to be wireheaded” is not a valid enough justification for not wireheading them in secret (for comparison, consider an angry teenager who break up with their partner and firmly says they want to immediately die.. you are probably going to ignore the stated preference and preventing them from suicide, knowing that this will result in much higher future happiness).
Only if they can see through the illusion in the first place
Well, yes, but the question is what humanity wants. Humanity does not want that to be done to it. It’s not what CEV would output, and if you’re in the position to implement a wrapper-mind with some utility function of your choice, putting that in it would be a violation of human preferences on a cosmological scale. That’s very much not leaving your fingerprints off the future.
consider an angry teenager who break up with their partner and firmly says they want to immediately die
That’s probably not what they’d actually want on reflection (i. e., if you ran the value conflict resolution and value extrapolation steps), just what they want in the moment.
“Humanity” is not a single agent. It can have preferences only insofar as the individual humans share similar preferences. If you are a happiness maximizer, for every individual human you look at, you’ll probably find that their happiness would be maximized by wireheading (because most whimsical desires like becoming the king of the world are not feasible to satisfy otherwise).
I’m not even that sure that CEV would avoid this problem. In which way being enclosed in a perfect world is not the best thing for you? Because it would be fake? But how do you know that’s fake? Imagine that an AGI offers to teleport you on another planet, which is perfectly suitable for you; you’ll land there and thrive forever. Now imagine that instead of actually teleporting you to another planet, it just let you to believe the same; you’ll (mentally) land there and thrive forever anyway. I mean, your brain is experiencing the very same thing! It’s not obvious that the second option is worse than the first one, unless you have an hardcoded limit like “Thou Shalt Not Wirehead People”.
Only if they can see through the illusion in the first place. Suppose that the happiness-maximizing AGI could throw you into a perfect fake reality without you even noticing. One day you wake up, and from that point on everything goes perfectly well for you in some very plausible manner. After a while, your happiness has reached immensely high levels. “This poor schmuck doesn’t want to be wireheaded” is not a valid enough justification for not wireheading them in secret (for comparison, consider an angry teenager who break up with their partner and firmly says they want to immediately die.. you are probably going to ignore the stated preference and preventing them from suicide, knowing that this will result in much higher future happiness).
Well, yes, but the question is what humanity wants. Humanity does not want that to be done to it. It’s not what CEV would output, and if you’re in the position to implement a wrapper-mind with some utility function of your choice, putting that in it would be a violation of human preferences on a cosmological scale. That’s very much not leaving your fingerprints off the future.
That’s probably not what they’d actually want on reflection (i. e., if you ran the value conflict resolution and value extrapolation steps), just what they want in the moment.
“Humanity” is not a single agent. It can have preferences only insofar as the individual humans share similar preferences. If you are a happiness maximizer, for every individual human you look at, you’ll probably find that their happiness would be maximized by wireheading (because most whimsical desires like becoming the king of the world are not feasible to satisfy otherwise).
I’m not even that sure that CEV would avoid this problem. In which way being enclosed in a perfect world is not the best thing for you? Because it would be fake? But how do you know that’s fake? Imagine that an AGI offers to teleport you on another planet, which is perfectly suitable for you; you’ll land there and thrive forever. Now imagine that instead of actually teleporting you to another planet, it just let you to believe the same; you’ll (mentally) land there and thrive forever anyway. I mean, your brain is experiencing the very same thing! It’s not obvious that the second option is worse than the first one, unless you have an hardcoded limit like “Thou Shalt Not Wirehead People”.