I’m not sure it makes sense to talk about what somebody’s values are in the absence of social pressure: people’s values are defined and shaped by those around them.
I’m also not convinced that every horrible thing people have ever done to the “outgroup” is motivated by fear. Oftentimes it is motivated by opportunistic selfishness taking advantage of broader societal apathy, like the slaveowners who sexually abused their slaves. Or just a deep-seated need to feel powerful and on top. There will always be some segment of society who wants somebody to be beneath them in the pecking order, and a much larger segment of society that doesn’t really care if that is the case as long as it isn’t them underneath. Anything else requires some kind of overwhelming utopian political victory that I don’t find likely.
If the aligned AI leaves anybody out of its consideration whatsoever, it will screw them over badly by maximizing the values of those among us who would exploit them. After all, if you don’t consider slaves people, the argument that we need to preserve the slaveowners’ freedom starts to make sense.
There are just so many excuses for suffering out there, and I don’t believe that the powers that be will shake off all of them in the next few decades. Here are a few examples:
They are a bad person, they deserve it
They are a nonbeliever, they deserve hell
It’s the natural order for us to rule over them
Suffering is necessary for meaning in life
Wild animal suffering is a beautiful part of the ecosystem which must be preserved
I’m not sure it makes sense to talk about what somebody’s values are in the absence of social pressure: people’s values are defined and shaped by those around them.
I’m also not convinced that every horrible thing people have ever done to the “outgroup” is motivated by fear. Oftentimes it is motivated by opportunistic selfishness taking advantage of broader societal apathy, like the slaveowners who sexually abused their slaves. Or just a deep-seated need to feel powerful and on top. There will always be some segment of society who wants somebody to be beneath them in the pecking order, and a much larger segment of society that doesn’t really care if that is the case as long as it isn’t them underneath. Anything else requires some kind of overwhelming utopian political victory that I don’t find likely.
If the aligned AI leaves anybody out of its consideration whatsoever, it will screw them over badly by maximizing the values of those among us who would exploit them. After all, if you don’t consider slaves people, the argument that we need to preserve the slaveowners’ freedom starts to make sense.
There are just so many excuses for suffering out there, and I don’t believe that the powers that be will shake off all of them in the next few decades. Here are a few examples:
They are a bad person, they deserve it
They are a nonbeliever, they deserve hell
It’s the natural order for us to rule over them
Suffering is necessary for meaning in life
Wild animal suffering is a beautiful part of the ecosystem which must be preserved