If I am completely and consistently aware of what I actually value, then yes, my desires are equivalent to my values and it makes no sense to talk about satisfying one while challenging the other (modulo cases of values-in-tension, as you say, which isn’t what I’m talking about).
My experience is that people are not completely and consistently aware of what they actually value, and it would astonish me if I turned out to be the fortunate exception.
Humans frequently treat instrumental goals as though they were terminal. Indeed, I suspect that’s all we ever do.
But even if I’m wrong, and it turns out that there really are terminal values in there somewhere, then I expect that most humans aren’t aware of them and if some external system starts optimizing for them, and is willing to trade arbitrary amounts of a merely instrumental good in exchange for the terminal good it serves as a proxy for (as well it should), we’ll experience that as emotionally unpleasant and challenging.
If I am completely and consistently aware of what I actually value, then yes, my desires are equivalent to my values and it makes no sense to talk about satisfying one while challenging the other (modulo cases of values-in-tension, as you say, which isn’t what I’m talking about).
My experience is that people are not completely and consistently aware of what they actually value, and it would astonish me if I turned out to be the fortunate exception.
Humans frequently treat instrumental goals as though they were terminal. Indeed, I suspect that’s all we ever do.
But even if I’m wrong, and it turns out that there really are terminal values in there somewhere, then I expect that most humans aren’t aware of them and if some external system starts optimizing for them, and is willing to trade arbitrary amounts of a merely instrumental good in exchange for the terminal good it serves as a proxy for (as well it should), we’ll experience that as emotionally unpleasant and challenging.
Solid answer, as far as I can see right now.