I would argue that any humans that had this bug in their utility function have (mostly) failed to reproduce, which is why most existing humans are opposed to wireheading.
Why would evolution come up with a fully general solution against such ‘bugs in our utility functions’?
Take addiction to a substance X. Evolution wouldn’t give us a psychological capacity to inspect our utility functions and to guard against such counterfeit utility. It would simply give us a distaste for substance X.
My guess is that we have some kind of self-referential utility function. We do not only want what our utility functions tell us we want. We also want utility (happiness) per se. And this want is itself included in that utility function!
When thinking about wireheading I think we are judging a tradeoff, between satisfying mere happiness and the states of affairs which we prefer (not including happiness).
So, people who have a strong component of “just be happy” in their utility function might choose to wirehead, and people in which other components are dominant might choose not to.
Why would evolution come up with a fully general solution against such ‘bugs in our utility functions’?
Take addiction to a substance X. Evolution wouldn’t give us a psychological capacity to inspect our utility functions and to guard against such counterfeit utility. It would simply give us a distaste for substance X.
My guess is that we have some kind of self-referential utility function. We do not only want what our utility functions tell us we want. We also want utility (happiness) per se. And this want is itself included in that utility function!
When thinking about wireheading I think we are judging a tradeoff, between satisfying mere happiness and the states of affairs which we prefer (not including happiness).
So, people who have a strong component of “just be happy” in their utility function might choose to wirehead, and people in which other components are dominant might choose not to.
That sounds reasonable.