Is an utopia that’d be perfect for everyone possible?
The short and obvious answer is no. Our civilization contains omnicidal maniacs and true sadists, whose central preferences are directly at odds with the preferences of most other people. Their happiness is diametrically opposed to other people’s.
If you are constrained to keep omnicidal maniacs in the same world as everyone else, this is obviously true.
But it doesn’t seem to be obviously true in every possible future. Imagine a world where every single sentient mind is wireheaded into their own personal paradise, interacting only with figments designed to maximize their preferences.
Some people might (under reflection) be locally-caring entities, but most people’s preferences are about what the reality actually contains and they (even under reflection) wouldn’t want to, for example, press a button that cause them to mistakenly believe that everything is fine.
Being embedded in a fake reality and fooled into believing it’s true would be against many people’s preferences.
Only if they can see through the illusion in the first place. Suppose that the happiness-maximizing AGI could throw you into a perfect fake reality without you even noticing. One day you wake up, and from that point on everything goes perfectly well for you in some very plausible manner. After a while, your happiness has reached immensely high levels. “This poor schmuck doesn’t want to be wireheaded” is not a valid enough justification for not wireheading them in secret (for comparison, consider an angry teenager who break up with their partner and firmly says they want to immediately die.. you are probably going to ignore the stated preference and preventing them from suicide, knowing that this will result in much higher future happiness).
Only if they can see through the illusion in the first place
Well, yes, but the question is what humanity wants. Humanity does not want that to be done to it. It’s not what CEV would output, and if you’re in the position to implement a wrapper-mind with some utility function of your choice, putting that in it would be a violation of human preferences on a cosmological scale. That’s very much not leaving your fingerprints off the future.
consider an angry teenager who break up with their partner and firmly says they want to immediately die
That’s probably not what they’d actually want on reflection (i. e., if you ran the value conflict resolution and value extrapolation steps), just what they want in the moment.
“Humanity” is not a single agent. It can have preferences only insofar as the individual humans share similar preferences. If you are a happiness maximizer, for every individual human you look at, you’ll probably find that their happiness would be maximized by wireheading (because most whimsical desires like becoming the king of the world are not feasible to satisfy otherwise).
I’m not even that sure that CEV would avoid this problem. In which way being enclosed in a perfect world is not the best thing for you? Because it would be fake? But how do you know that’s fake? Imagine that an AGI offers to teleport you on another planet, which is perfectly suitable for you; you’ll land there and thrive forever. Now imagine that instead of actually teleporting you to another planet, it just let you to believe the same; you’ll (mentally) land there and thrive forever anyway. I mean, your brain is experiencing the very same thing! It’s not obvious that the second option is worse than the first one, unless you have an hardcoded limit like “Thou Shalt Not Wirehead People”.
If you are constrained to keep omnicidal maniacs in the same world as everyone else, this is obviously true.
But it doesn’t seem to be obviously true in every possible future. Imagine a world where every single sentient mind is wireheaded into their own personal paradise, interacting only with figments designed to maximize their preferences.
Being embedded in a fake reality and fooled into believing it’s true would be against many people’s preferences.
(I suppose you can imagine some galaxy-brained ideas for emigration into different regions of Tegmark IV, Permutation City-style?)
Strongly agree; I have an old, short post about this. See also Contact with reality.
Some people might (under reflection) be locally-caring entities, but most people’s preferences are about what the reality actually contains and they (even under reflection) wouldn’t want to, for example, press a button that cause them to mistakenly believe that everything is fine.
Only if they can see through the illusion in the first place. Suppose that the happiness-maximizing AGI could throw you into a perfect fake reality without you even noticing. One day you wake up, and from that point on everything goes perfectly well for you in some very plausible manner. After a while, your happiness has reached immensely high levels. “This poor schmuck doesn’t want to be wireheaded” is not a valid enough justification for not wireheading them in secret (for comparison, consider an angry teenager who break up with their partner and firmly says they want to immediately die.. you are probably going to ignore the stated preference and preventing them from suicide, knowing that this will result in much higher future happiness).
Well, yes, but the question is what humanity wants. Humanity does not want that to be done to it. It’s not what CEV would output, and if you’re in the position to implement a wrapper-mind with some utility function of your choice, putting that in it would be a violation of human preferences on a cosmological scale. That’s very much not leaving your fingerprints off the future.
That’s probably not what they’d actually want on reflection (i. e., if you ran the value conflict resolution and value extrapolation steps), just what they want in the moment.
“Humanity” is not a single agent. It can have preferences only insofar as the individual humans share similar preferences. If you are a happiness maximizer, for every individual human you look at, you’ll probably find that their happiness would be maximized by wireheading (because most whimsical desires like becoming the king of the world are not feasible to satisfy otherwise).
I’m not even that sure that CEV would avoid this problem. In which way being enclosed in a perfect world is not the best thing for you? Because it would be fake? But how do you know that’s fake? Imagine that an AGI offers to teleport you on another planet, which is perfectly suitable for you; you’ll land there and thrive forever. Now imagine that instead of actually teleporting you to another planet, it just let you to believe the same; you’ll (mentally) land there and thrive forever anyway. I mean, your brain is experiencing the very same thing! It’s not obvious that the second option is worse than the first one, unless you have an hardcoded limit like “Thou Shalt Not Wirehead People”.