Also, even if we average over, or coherently extrapolate, the combined opinions of all humans, that can and will change over time. Especially once we have effective genetic engineering of human ethical instincts, or are cyborging heavily, or even just have ASIs with superhuman levels of persuasion (if that turns out to be a thing), this could then change a lot, making the contents of that far-future light-cone entirely unpredictable and almost certainly very alien to us now. A problem I discuss at length in The Mutable Values Problem in Value Learning and CEV.
Completely agreed.
Also, even if we average over, or coherently extrapolate, the combined opinions of all humans, that can and will change over time. Especially once we have effective genetic engineering of human ethical instincts, or are cyborging heavily, or even just have ASIs with superhuman levels of persuasion (if that turns out to be a thing), this could then change a lot, making the contents of that far-future light-cone entirely unpredictable and almost certainly very alien to us now. A problem I discuss at length in The Mutable Values Problem in Value Learning and CEV.