“Humanity” is not a single agent. It can have preferences only insofar as the individual humans share similar preferences. If you are a happiness maximizer, for every individual human you look at, you’ll probably find that their happiness would be maximized by wireheading (because most whimsical desires like becoming the king of the world are not feasible to satisfy otherwise).
I’m not even that sure that CEV would avoid this problem. In which way being enclosed in a perfect world is not the best thing for you? Because it would be fake? But how do you know that’s fake? Imagine that an AGI offers to teleport you on another planet, which is perfectly suitable for you; you’ll land there and thrive forever. Now imagine that instead of actually teleporting you to another planet, it just let you to believe the same; you’ll (mentally) land there and thrive forever anyway. I mean, your brain is experiencing the very same thing! It’s not obvious that the second option is worse than the first one, unless you have an hardcoded limit like “Thou Shalt Not Wirehead People”.
“Humanity” is not a single agent. It can have preferences only insofar as the individual humans share similar preferences. If you are a happiness maximizer, for every individual human you look at, you’ll probably find that their happiness would be maximized by wireheading (because most whimsical desires like becoming the king of the world are not feasible to satisfy otherwise).
I’m not even that sure that CEV would avoid this problem. In which way being enclosed in a perfect world is not the best thing for you? Because it would be fake? But how do you know that’s fake? Imagine that an AGI offers to teleport you on another planet, which is perfectly suitable for you; you’ll land there and thrive forever. Now imagine that instead of actually teleporting you to another planet, it just let you to believe the same; you’ll (mentally) land there and thrive forever anyway. I mean, your brain is experiencing the very same thing! It’s not obvious that the second option is worse than the first one, unless you have an hardcoded limit like “Thou Shalt Not Wirehead People”.