Not to evoke a recursive nightmare, but some utility function alterations appear to be strictly desirable.
As an obvious example, if I were on a diet and I could rewrite my utility function such that the utilities assigned to consuming spinach and cheesecake were swapped, I see no harm in making that edit. One could argue that my second-order utility (and all higher) function should be collapsed into my first-order one, such that this would not really change my meta-utility function, but this issue just highlights the futility of trying to cram my complex, conflicting, and oft-inconsistent desires into a utility function.
This does raise an interesting issue: if I’m a strictly selfish utilitarian, do I not want my utility function to be that which will attain the highest expected utility? Selfishness is not necessary; it just makes the question much simpler.
I wouldn’t claim that any human is actually able to describe their own utility function; they’re much too complex and riddled with strange exceptions and pieces of craziness like hyperbolic discounting.
I also think that there’s some confusion surrounding the whole idea of utility functions in reality, which I should have been more explicit about. Your utility function is just a description of what you want/value; it is not explicitly about maximizing happiness. For example, I don’t want to murder people, even under circumstances where it would make me very happy to do so. For this reason, I would do everything within my power to avoid taking a pill that would change my preferences such that I would then generally want to murder people; this is the murder pill I mentioned.
As for swapping the utilities of spinach and cheesecake, I think the only way that makes sense to do so would be to change how you perceive their respective tastes, which isn’t a change to your utility function at all. You still want to eat food that tastes good; changing that would have much broader and less predictable consequences.
This does raise an interesting issue: if I’m a strictly selfish utilitarian, do I not want my utility function to be that which will attain the highest expected utility? Selfishness is not necessary; it just makes the question much simpler.
Only if your current utility function is “maximize expected utility.” (It isn’t.)
This does raise an interesting issue: if I’m a strictly selfish utilitarian, do I not want my utility function to be that which will attain the highest expected utility?
This is a particular form of wireheading; fortunately, for evolutionary reasons we’re not able to do very much of it without advanced technology.
This does raise an interesting issue: if I’m a strictly selfish utilitarian, do I not want my utility function to be that which will attain the highest expected utility?
This is a particular form of wireheading
I’d say it’s rather a form of conceptual confusion: you can’t change a concept (“change” is itself a “timeful” concept, meaningful only as a property within structures which are processes in the appropriate sense). But it’s plausible that creating agents with slightly different explicit preference will result in a better outcome than, all else equal, if you give those agents your own preference. Of course, you’d probably need to be a superintelligence to correctly make decisions like this, at which point creation of agents with given preference might cease to be a natural concept.
Not to evoke a recursive nightmare, but some utility function alterations appear to be strictly desirable.
As an obvious example, if I were on a diet and I could rewrite my utility function such that the utilities assigned to consuming spinach and cheesecake were swapped, I see no harm in making that edit. One could argue that my second-order utility (and all higher) function should be collapsed into my first-order one, such that this would not really change my meta-utility function, but this issue just highlights the futility of trying to cram my complex, conflicting, and oft-inconsistent desires into a utility function.
This does raise an interesting issue: if I’m a strictly selfish utilitarian, do I not want my utility function to be that which will attain the highest expected utility? Selfishness is not necessary; it just makes the question much simpler.
I wouldn’t claim that any human is actually able to describe their own utility function; they’re much too complex and riddled with strange exceptions and pieces of craziness like hyperbolic discounting.
I also think that there’s some confusion surrounding the whole idea of utility functions in reality, which I should have been more explicit about. Your utility function is just a description of what you want/value; it is not explicitly about maximizing happiness. For example, I don’t want to murder people, even under circumstances where it would make me very happy to do so. For this reason, I would do everything within my power to avoid taking a pill that would change my preferences such that I would then generally want to murder people; this is the murder pill I mentioned.
As for swapping the utilities of spinach and cheesecake, I think the only way that makes sense to do so would be to change how you perceive their respective tastes, which isn’t a change to your utility function at all. You still want to eat food that tastes good; changing that would have much broader and less predictable consequences.
Only if your current utility function is “maximize expected utility.” (It isn’t.)
Anorexia could be viewed as an excessive ability to rewrite utility functions about food.
If you don’t have the ability to include context, the biological blind god may serve you better than the memetic blind god.
This is a particular form of wireheading; fortunately, for evolutionary reasons we’re not able to do very much of it without advanced technology.
I’d say it’s rather a form of conceptual confusion: you can’t change a concept (“change” is itself a “timeful” concept, meaningful only as a property within structures which are processes in the appropriate sense). But it’s plausible that creating agents with slightly different explicit preference will result in a better outcome than, all else equal, if you give those agents your own preference. Of course, you’d probably need to be a superintelligence to correctly make decisions like this, at which point creation of agents with given preference might cease to be a natural concept.
I am afraid that advanced technology is not necessary. Literal wireheading.