If a majority of humanity wishes to kill a minority, obviously there won’t be a consensus to stop the killing, and AI will not interfere.
This assumes that CEV uses something along the lines of a simulated vote as an aggregation mechanism. Currently the method of aggregation is undefined so we can’t say this with confidence—certainly not as something obvious.
I agree. However, if the CEV doesn’t privilege any value separately from how many people value it how much (in EV), and if the EV of a large majority values killing a small minority (whose EV is of course opposed), and if you have protection against both positive and negative utility monsters (so it’s at least not obvious and automatic that the negative value of the minority would outweigh the positive value of the majority) - then my scenario seems to me to be plausible, and an explanation is necessary as to how it might be prevented.
Of course you could say that until CEV is really formally specified, and we know how the aggregation works, this explanation cannot be produced.
This assumes that CEV uses something along the lines of a simulated vote as an aggregation mechanism. Currently the method of aggregation is undefined so we can’t say this with confidence—certainly not as something obvious.
I agree. However, if the CEV doesn’t privilege any value separately from how many people value it how much (in EV), and if the EV of a large majority values killing a small minority (whose EV is of course opposed), and if you have protection against both positive and negative utility monsters (so it’s at least not obvious and automatic that the negative value of the minority would outweigh the positive value of the majority) - then my scenario seems to me to be plausible, and an explanation is necessary as to how it might be prevented.
Of course you could say that until CEV is really formally specified, and we know how the aggregation works, this explanation cannot be produced.
Absolutely, on both counts.