Coherence is not a simple question of a majority vote. Coherence will reflect the balance, concentration, and strength of individual volitions. A minor, muddled preference of 60% of humanity might be countered by a strong, unmuddled preference of 10% of humanity. The variables are quantitative, not qualitative.
(...)
It should be easier to counter coherence than to create coherence.
(...)
In qualitative terms, our unimaginably alien, powerful, and humane future selves should have a strong ability to say “Wait! Stop! You’re going to predictably regret that!”, but we should require much higher standards of predictability and coherence before we trust the extrapolation that says “Do this specific positive thing, even if you can’t comprehend why.”
Though it’s not clear to me how the document would deal with Wei Dai’s point in the sibling comment. In the absence of coherence on the question of whether to protect, persecute, or ignore impopular minority groups, does CEV default to protecting them or ignoring them? You might say that as written, it would obviously not protect them, because there was no coherence in favor of doing so; but what if protection of minority groups is a side effect of other measures CEV was taking anyway?
(For what it’s worth, I suspect that extrapolation would in fact create enough coherence for this particular scenario not to be a problem.)
My understanding is that CEV is based on consensus, in which case the majority is meaningless.
If CEV doesn’t positively value some minority group not being killed (i.e., if it’s just indifferent due to not having a consensus), then the majority would be free to try to kill that group. So we really do need CEV to saying something about this, instead of nothing.
My understanding is that CEV is based on consensus, in which case the majority is meaningless.
Some quotes from the CEV document:
Though it’s not clear to me how the document would deal with Wei Dai’s point in the sibling comment. In the absence of coherence on the question of whether to protect, persecute, or ignore impopular minority groups, does CEV default to protecting them or ignoring them? You might say that as written, it would obviously not protect them, because there was no coherence in favor of doing so; but what if protection of minority groups is a side effect of other measures CEV was taking anyway?
(For what it’s worth, I suspect that extrapolation would in fact create enough coherence for this particular scenario not to be a problem.)
Thank you. So, not quite consensus but similarly biased in favor if inaction.
If CEV doesn’t positively value some minority group not being killed (i.e., if it’s just indifferent due to not having a consensus), then the majority would be free to try to kill that group. So we really do need CEV to saying something about this, instead of nothing.
Assuming we have no other checks on behavior, yes. I’m not sure, pending more reflection, whether that’s a fair assumption or not...
There is absolutely no reason to think that the values of all humans, extrapolated in some way, will arrive at a consensus.