I’m generally of the opinion that CEV was always a bad goal, and that we shouldn’t attempt to do so, and a big reason for this is I don’t believe a procedure exists that doesn’t incentivize humans to fight over the initial dynamic, or another way to say this is that who implements the CEV procedure will always matter, because I don’t believe in the idea that humans will naturally converge in the limit of more intelligence to a fixed moral value system, and instead I predict divergence as constraints are removed.
I roughly agree with Steven Byrnes, but stronger here (though I think this holds beyond humans too):
Regardless, imo the biggest question that standard CEV leaves unanswered is what your starting population looks like that you extrapolate from. The obvious answer is “all the currently living humans,” but I find that to be a very unsatisfying answer. One of the principles that Eliezer talks about in discussing CEV is that you want a procedure such that it doesn’t matter who implements it—see Eliezer’s discussion under “Avoid creating a motive for modern-day humans to fight over the initial dynamic.” I think this is a great principle, but imo it doesn’t go far enough. In particular:
I’m generally of the opinion that CEV was always a bad goal, and that we shouldn’t attempt to do so, and a big reason for this is I don’t believe a procedure exists that doesn’t incentivize humans to fight over the initial dynamic, or another way to say this is that who implements the CEV procedure will always matter, because I don’t believe in the idea that humans will naturally converge in the limit of more intelligence to a fixed moral value system, and instead I predict divergence as constraints are removed.
I roughly agree with Steven Byrnes, but stronger here (though I think this holds beyond humans too):
https://www.lesswrong.com/posts/SqgRtCwueovvwxpDQ/valence-series-2-valence-and-normativity#2_7_3_Possible_implications_for_AI_alignment_discourse