Very idealistic. What if putting down other humans is an actual terminal value for some people?
Something in their mind would most likely be broken for this to happen, according to data I’ve seen. Even the worst polypaths (sociopath + psychopath + whatever pathy you want to throw in for the most despicable example of human possible) apparently still see themselves as heroes “saving the world”, or at least as some kind of “good guy in the long run”. Not that I’m implying that the “All Evil people are actually Good because they just know that the world needs balance and humans need a common enemy” myth is true, because that’s been shown false even more clearly.
More importantly, CEV aims are reflectively coherent values. If they have a terminal value of killing people, this value is extremely likely to conflict with other of their values, maybe even with their own wishes regarding self-values (“I wish I didn’t enjoy killing humans so much”), and would definitely not cohere with most other humans’ values unless there’s four billion people out there who secretly desire very very much to kill humans all the time but live in utter misery thanks to a global conspiracy that successfully chains them down or some other just-as-unlikely factor (e.g. some undiscovered freeloader’s problem or tragedy of the commons that we’ve somehow never noticed).
Overall, my current odds are very very low that any such value would survive extrapolation when you attempt to have a reflectively coherent system where someone wishes they didn’t value X, but do value it.
Something in their mind would most likely be broken for this to happen, according to data I’ve seen. Even the worst polypaths (sociopath + psychopath + whatever pathy you want to throw in for the most despicable example of human possible) apparently still see themselves as heroes “saving the world”, or at least as some kind of “good guy in the long run”. Not that I’m implying that the “All Evil people are actually Good because they just know that the world needs balance and humans need a common enemy” myth is true, because that’s been shown false even more clearly.
More importantly, CEV aims are reflectively coherent values. If they have a terminal value of killing people, this value is extremely likely to conflict with other of their values, maybe even with their own wishes regarding self-values (“I wish I didn’t enjoy killing humans so much”), and would definitely not cohere with most other humans’ values unless there’s four billion people out there who secretly desire very very much to kill humans all the time but live in utter misery thanks to a global conspiracy that successfully chains them down or some other just-as-unlikely factor (e.g. some undiscovered freeloader’s problem or tragedy of the commons that we’ve somehow never noticed).
Overall, my current odds are very very low that any such value would survive extrapolation when you attempt to have a reflectively coherent system where someone wishes they didn’t value X, but do value it.