Why is it a terrible idea? Imagine that our ancestors thought that regular human sacrifices to God of Rain are required for societal survival, and it would be “especially heinous” to doom the society by abandoning this practice, so they decided to “lock in” this value. We have a lot of these grandfathered values that no longer make sense already locked in, intentionally or accidentally.
It would be terrible by our values. Sure. Would it be terrible by their values? That is more complicated. If they are arguing it is required for “social survival”, then that sound like they were mistaken on a purely factual question. They failed to trace their values back to the source. They should have locked in a value for “social survival”. And then any factual beliefs about the correlation between human sacrifice to rain gods and social survival are updated with normal baysian updates.
But let’s suppose they truely deeply valued human sacrifice. Not just for the sake of something else, but for it’s own sake. Then their mind and yours have a fundamental disagreement. Neither of you will persuade the other of your values.
If values aren’t locked in, they drift. What phenomena cause that drift? If our ancestors can have truely terrible values (by our values), our decedents can be just as bad. So you refuse to lock in your values, and 500 years later, a bunch of people who value human sacrifice decide to lock in their values. Or maybe you lock in the meta value of no one having the power to lock in their object values, and values drift until the end of the universe. Value space is large, and 99% of the values it drifts through would be horrible as measured by your current values.
It would be terrible by our values. Sure. Would it be terrible by their values? That is more complicated. If they are arguing it is required for “social survival”, then that sound like they were mistaken on a purely factual question. They failed to trace their values back to the source. They should have locked in a value for “social survival”. And then any factual beliefs about the correlation between human sacrifice to rain gods and social survival are updated with normal baysian updates.
But let’s suppose they truely deeply valued human sacrifice. Not just for the sake of something else, but for it’s own sake. Then their mind and yours have a fundamental disagreement. Neither of you will persuade the other of your values.
If values aren’t locked in, they drift. What phenomena cause that drift? If our ancestors can have truely terrible values (by our values), our decedents can be just as bad. So you refuse to lock in your values, and 500 years later, a bunch of people who value human sacrifice decide to lock in their values. Or maybe you lock in the meta value of no one having the power to lock in their object values, and values drift until the end of the universe. Value space is large, and 99% of the values it drifts through would be horrible as measured by your current values.