Imagine someone offers you an extremely high-paying job. Unfortunately, the job involves something you find morally repulsive – say, child trafficking. But the recruiter offers you a pill that will rewrite your brain chemistry so that you’ll no longer find it repulsive. Would you take the pill?
I think that pill would reasonably be categorized as “updating your goals”. If you take it, you can then accept the lucrative job and presumably you’ll be well positioned to satisfy your new/remaining goals, i.e. you’ll be “happy”. But you’d be acting against your pre-pill goal (I am glossing over exactly what that goal is, perhaps “not harming children” although I’m sure there’s more to unpack here).
I pose this example in an attempt to get at the heart of “distinguishing between terminal and instrumental goals” as suggested by quetzal_rainbow. This is also my intuition, that it’s a question of terminal vs. instrumental goals.
Imagine someone offers you an extremely high-paying job. Unfortunately, the job involves something you find morally repulsive – say, child trafficking. But the recruiter offers you a pill that will rewrite your brain chemistry so that you’ll no longer find it repulsive. Would you take the pill?
I think that pill would reasonably be categorized as “updating your goals”. If you take it, you can then accept the lucrative job and presumably you’ll be well positioned to satisfy your new/remaining goals, i.e. you’ll be “happy”. But you’d be acting against your pre-pill goal (I am glossing over exactly what that goal is, perhaps “not harming children” although I’m sure there’s more to unpack here).
I pose this example in an attempt to get at the heart of “distinguishing between terminal and instrumental goals” as suggested by quetzal_rainbow. This is also my intuition, that it’s a question of terminal vs. instrumental goals.