There is no “correct” way whatsoever in setting your terminal values, your “ultimate goals” (other agents may prefer you to pursue values similar to their own, whatever those may be). Your ultimate goals can include anything from “maximize the number of paperclips” to “paint everything blue” to “always keep in a state of being nourished (for the sake of itself!)” or “always keep in a state of emotional fulfillment through short-term altruistic deeds”.
Based on those ultimate goals, you define other, derivative goals, such as “I want to buy blue paint” as an intermediate goal towards “so I can paint everything blue”. Those “stepping stones” can be irrational / insignificant (in relation to pursuing your terminal values), i.e. you can be “wrong” about them. Maybe you shouldn’t buy blue paint, but rather produce it yourself. Or rather invest in nanotechnology to paint everything blue using nanomagic.
Only you can (or can’t, humans are notoriously bad at accurately providing their actual utility functions) try to elucidate what your ultimate goals are, but having decided on them, they are supra-rational / beyond rational / ‘rational not applicable’ by definition.
There is no fault in choosing “I want to live a life that maximizes fuzzy feelings through charitable acts” over “I’m dedicating my life to decreasing the Gini index, whatever the personal cost to myself.”
There is no “correct” way whatsoever in setting your terminal values, your “ultimate goals” (other agents may prefer you to pursue values similar to their own, whatever those may be). Your ultimate goals can include anything from “maximize the number of paperclips” to “paint everything blue” to “always keep in a state of being nourished (for the sake of itself!)” or “always keep in a state of emotional fulfillment through short-term altruistic deeds”.
Based on those ultimate goals, you define other, derivative goals, such as “I want to buy blue paint” as an intermediate goal towards “so I can paint everything blue”. Those “stepping stones” can be irrational / insignificant (in relation to pursuing your terminal values), i.e. you can be “wrong” about them. Maybe you shouldn’t buy blue paint, but rather produce it yourself. Or rather invest in nanotechnology to paint everything blue using nanomagic.
Only you can (or can’t, humans are notoriously bad at accurately providing their actual utility functions) try to elucidate what your ultimate goals are, but having decided on them, they are supra-rational / beyond rational / ‘rational not applicable’ by definition.
There is no fault in choosing “I want to live a life that maximizes fuzzy feelings through charitable acts” over “I’m dedicating my life to decreasing the Gini index, whatever the personal cost to myself.”