I would distinguish several levels of meta-preferences.
On level 1, an agent has a set of object-level preferences, and wants to achieve the maximum cumulative satisfaction of them over the lifetime. To do that, the agent may want sometimes to override the incentive to maximize the satisfaction at each step if it is harmful in the long run. Basically, it’s just switching from a greedy gradient descent to something smarter, and barely requires any manipulations with object-level preferences.
On level 2, the agent may want to change their set of object-level preferences in order to achieve higher satisfaction, given the realistic limits of what’s possible. A stupid example: someone who wants one billion dollars but cannot have it may want to start wanting ten dollars instead, and be much happier. More realistic example: a person who became disables may want to readjust their preference and accommodate new limitations. Applying this strategy to its logical end has some failure modes (e.g. the one described in Three Worlds Collide, or, more trivially, opiates), but it still sort of make sense for a utility-driven agent.
On level 3, the agent may want to add or remove some preferences, regardless of the effect of that on the total level of satisfaction, just for their own sake.
Wanting to care more about animals seems to be level-3 meta-preference. In a world where this preference is horribly dissatisfied, where animals are killed at the rate of about one kiloholocaust per year, that clearly doesn’t optimize for satisfaction. Consistency of values and motivations—yes, but only if you happen to have consistency as a terminal value in the utility function. That doesn’t necessarily have to be the case: in most scenarios, consistency is good because it’s useful, because it allows us to solve problems. The lack of compassion to animals doesn’t seem to be a problem, unless the inconsistency itself is a problem.
Thus, it seems impossible to make such change without accepting that carrying about animals is good or that having consistent values is good in a morally realist way. Now, I’m not claiming that I’m a complete moral relativist. I’m not even sure that it’s possible—so far, all the arguments for moral relativism I’ve seen are actually realist themselves. However, arguing for switching between different realist-ish moral frameworks seems to be a much harder task.
I would distinguish several levels of meta-preferences.
On level 1, an agent has a set of object-level preferences, and wants to achieve the maximum cumulative satisfaction of them over the lifetime. To do that, the agent may want sometimes to override the incentive to maximize the satisfaction at each step if it is harmful in the long run. Basically, it’s just switching from a greedy gradient descent to something smarter, and barely requires any manipulations with object-level preferences.
On level 2, the agent may want to change their set of object-level preferences in order to achieve higher satisfaction, given the realistic limits of what’s possible. A stupid example: someone who wants one billion dollars but cannot have it may want to start wanting ten dollars instead, and be much happier. More realistic example: a person who became disables may want to readjust their preference and accommodate new limitations. Applying this strategy to its logical end has some failure modes (e.g. the one described in Three Worlds Collide, or, more trivially, opiates), but it still sort of make sense for a utility-driven agent.
On level 3, the agent may want to add or remove some preferences, regardless of the effect of that on the total level of satisfaction, just for their own sake.
Wanting to care more about animals seems to be level-3 meta-preference. In a world where this preference is horribly dissatisfied, where animals are killed at the rate of about one kiloholocaust per year, that clearly doesn’t optimize for satisfaction. Consistency of values and motivations—yes, but only if you happen to have consistency as a terminal value in the utility function. That doesn’t necessarily have to be the case: in most scenarios, consistency is good because it’s useful, because it allows us to solve problems. The lack of compassion to animals doesn’t seem to be a problem, unless the inconsistency itself is a problem.
Thus, it seems impossible to make such change without accepting that carrying about animals is good or that having consistent values is good in a morally realist way. Now, I’m not claiming that I’m a complete moral relativist. I’m not even sure that it’s possible—so far, all the arguments for moral relativism I’ve seen are actually realist themselves. However, arguing for switching between different realist-ish moral frameworks seems to be a much harder task.