There’s obviously no coherence if the terminal values of space-Jews include their continuing existence, and the terminal values of space-Nazis include the space-Jews’ eradication.
Prints out “these species’ values do not cohere”? Or perhaps “both species coherent-extrapolatedly appreciate pretty sunsets, therefore maximize prettiness of sunsets, but don’t do anything that impacts on the space-Jews survival one way or another, or the space-Nazis survival either if that connects negatively to the former?”
Note that the CEV must necessarily address contradicting terminal values. Thus an FAI is assumed to be powerful enough to affect people’s terminal values, at least over time.
For example, (some of the) Nazis might be OK with not wanting Jews dead, they are just unable to change their innate Jewphobia. An analogy would be people who are afraid of snakes but would not mind living in a world where snakes are non-poisonous (and not dangerous in any other way) and they are not afraid of them.
It would probably least-destructively turn the jews into nazis or vice versa; e.g. alter one or the other’s terminal values such that they were fully compatible. After all, if the only difference between jews and nazis is the nose, why not ask the jews to change the nose and gain an anti-former-nose preference (theoretically the jews would gain utility because they’d have a new terminal value they could satisfy). Of course this is a fine example of how meaningless terminal values can survive despite their innate meaningless; the nazis should realize the irrationality of their terminal value and simply drop it. But will CEV force them to drop it? Probably not. The practical effect is the dissolution of practical utility; utility earned from satisfying an anti-jew preference necessarily reduces the amount of utility attainable from other possible terminal values. That should be a strong argument CEV has to convince any group that one of their terminal values can be dropped, by comparing the opportunity cost of satisfying it to the benefit of satisfying other terminal values. This is even more of a digression from the original question, but I think this implies that CEV may eventually settle on a single, maximally effective terminal value.
I think CEV is supposed execute a controlled shutdown in that kind of situation and helpfully inform the operators that they live in a horrible, horrible world.
Inspired by this comment, here’s a question: what would the CEV of the inhabitants of shminux’s hypothetical world look like?
There’s obviously no coherence if the terminal values of space-Jews include their continuing existence, and the terminal values of space-Nazis include the space-Jews’ eradication.
So what does the algorithm do when you run it?
Prints out “these species’ values do not cohere”? Or perhaps “both species coherent-extrapolatedly appreciate pretty sunsets, therefore maximize prettiness of sunsets, but don’t do anything that impacts on the space-Jews survival one way or another, or the space-Nazis survival either if that connects negatively to the former?”
Return a “divide by zero”-type error, or send your Turing machine up in smoke trying.
Note that the CEV must necessarily address contradicting terminal values. Thus an FAI is assumed to be powerful enough to affect people’s terminal values, at least over time.
For example, (some of the) Nazis might be OK with not wanting Jews dead, they are just unable to change their innate Jewphobia. An analogy would be people who are afraid of snakes but would not mind living in a world where snakes are non-poisonous (and not dangerous in any other way) and they are not afraid of them.
It would probably least-destructively turn the jews into nazis or vice versa; e.g. alter one or the other’s terminal values such that they were fully compatible. After all, if the only difference between jews and nazis is the nose, why not ask the jews to change the nose and gain an anti-former-nose preference (theoretically the jews would gain utility because they’d have a new terminal value they could satisfy). Of course this is a fine example of how meaningless terminal values can survive despite their innate meaningless; the nazis should realize the irrationality of their terminal value and simply drop it. But will CEV force them to drop it? Probably not. The practical effect is the dissolution of practical utility; utility earned from satisfying an anti-jew preference necessarily reduces the amount of utility attainable from other possible terminal values. That should be a strong argument CEV has to convince any group that one of their terminal values can be dropped, by comparing the opportunity cost of satisfying it to the benefit of satisfying other terminal values. This is even more of a digression from the original question, but I think this implies that CEV may eventually settle on a single, maximally effective terminal value.
I think CEV is supposed execute a controlled shutdown in that kind of situation and helpfully inform the operators that they live in a horrible, horrible world.