To expand on my categorization of values a bit more, it seems clear to me that at least some human value do not deserved to be forever etched into the utility function of a singleton. Those caused by idiosyncratic environmental characteristics like taste for salt and sugar, for example. To me, these are simply accidents of history, and I wouldn’t hesitate (too much) to modify them away in myself, perhaps to be replaced by more interesting and exotic tastes.
What about reproduction? It’s a value that my genes programmed into me for their own purposes, so why should I be obligated to stick with it forever?
Or consider boredom. Eventually I may become so powerful that I can easily find the globally optimal course of action for any set of goals I might have, and notice that the optimal course of action often involves repetition of some kind. Why should I retain my desire not to do the same thing over and over again, which was programmed into me by evolution back when minds had a tendency to get stuck in local optimums?
And once I finally came to that realization, I felt less ashamed of values that seemed ‘provincial’ - but that’s another matter.
Eliezer, I wonder if this actually has more to do with your current belief that rationality equals expected utility maximization. For an expected utility maximizer, there is no distinction between ‘provincial’ and ‘universal’ values, and certainly no reason to ever feel ashamed of one’s values. One just optimizes according to whatever values one happens to have. But as I argued before, human beings are not expected utility maximizers, and I don’t see why we should try to emulate them, especially this aspect.
To expand on my categorization of values a bit more, it seems clear to me that at least some human value do not deserved to be forever etched into the utility function of a singleton. Those caused by idiosyncratic environmental characteristics like taste for salt and sugar, for example. To me, these are simply accidents of history, and I wouldn’t hesitate (too much) to modify them away in myself, perhaps to be replaced by more interesting and exotic tastes.
What about reproduction? It’s a value that my genes programmed into me for their own purposes, so why should I be obligated to stick with it forever?
Or consider boredom. Eventually I may become so powerful that I can easily find the globally optimal course of action for any set of goals I might have, and notice that the optimal course of action often involves repetition of some kind. Why should I retain my desire not to do the same thing over and over again, which was programmed into me by evolution back when minds had a tendency to get stuck in local optimums?
And once I finally came to that realization, I felt less ashamed of values that seemed ‘provincial’ - but that’s another matter.
Eliezer, I wonder if this actually has more to do with your current belief that rationality equals expected utility maximization. For an expected utility maximizer, there is no distinction between ‘provincial’ and ‘universal’ values, and certainly no reason to ever feel ashamed of one’s values. One just optimizes according to whatever values one happens to have. But as I argued before, human beings are not expected utility maximizers, and I don’t see why we should try to emulate them, especially this aspect.