I suspect that quite a few humans (including me) would be horrified by the actual implications of their own values being maximally realized.
How exactly could you be “horrified” about that unless you were comparing some of your own values being maximally realized with some of your other values not being maximally realized?
In other words, it doesn’t make sense (doesn’t even mean anything!) to say that you would be horrified (isn’t that a bad thing?) to have your desires fulfilled (isn’t that a good thing?), unless you’re really just talking about some of your desires conflicting with some of your other desires.
Because humans’ values are not a coherent, consistent set. We execute an evolutionarily-determined grab-bag of adaptations; there is no reason to assume this grab-bag adds up to a more coherent whole than “don’t die out.” (And even that’s a teleological projection onto stuff just happening.)
If I am completely and consistently aware of what I actually value, then yes, my desires are equivalent to my values and it makes no sense to talk about satisfying one while challenging the other (modulo cases of values-in-tension, as you say, which isn’t what I’m talking about).
My experience is that people are not completely and consistently aware of what they actually value, and it would astonish me if I turned out to be the fortunate exception.
Humans frequently treat instrumental goals as though they were terminal. Indeed, I suspect that’s all we ever do.
But even if I’m wrong, and it turns out that there really are terminal values in there somewhere, then I expect that most humans aren’t aware of them and if some external system starts optimizing for them, and is willing to trade arbitrary amounts of a merely instrumental good in exchange for the terminal good it serves as a proxy for (as well it should), we’ll experience that as emotionally unpleasant and challenging.
How exactly could you be “horrified” about that unless you were comparing some of your own values being maximally realized with some of your other values not being maximally realized?
In other words, it doesn’t make sense (doesn’t even mean anything!) to say that you would be horrified (isn’t that a bad thing?) to have your desires fulfilled (isn’t that a good thing?), unless you’re really just talking about some of your desires conflicting with some of your other desires.
Because humans’ values are not a coherent, consistent set. We execute an evolutionarily-determined grab-bag of adaptations; there is no reason to assume this grab-bag adds up to a more coherent whole than “don’t die out.” (And even that’s a teleological projection onto stuff just happening.)
If I am completely and consistently aware of what I actually value, then yes, my desires are equivalent to my values and it makes no sense to talk about satisfying one while challenging the other (modulo cases of values-in-tension, as you say, which isn’t what I’m talking about).
My experience is that people are not completely and consistently aware of what they actually value, and it would astonish me if I turned out to be the fortunate exception.
Humans frequently treat instrumental goals as though they were terminal. Indeed, I suspect that’s all we ever do.
But even if I’m wrong, and it turns out that there really are terminal values in there somewhere, then I expect that most humans aren’t aware of them and if some external system starts optimizing for them, and is willing to trade arbitrary amounts of a merely instrumental good in exchange for the terminal good it serves as a proxy for (as well it should), we’ll experience that as emotionally unpleasant and challenging.
Solid answer, as far as I can see right now.