It is a common thesis here that most humans would ultimately have the same moral judgments if they were in full agreement about all factual questions and were better at reasoning.
Maybe it’s true if you also specify “if they were fully capable of modifying their own moral intuitions.” I have an intuition (an unexamined belief? a hope? a sci-fi trope?) that humanity as a whole will continue to evolve morally and roughly converge on a morality that resembles current first-world liberal values more than, say, Old Testament values. That is, it would converge, in the limit of global prosperity and peace and dialogue, and assuming no singularity occurs and the average lifespan stays constant. You can call this naive if you want to; I don’t know whether it’s true. It’s what I imagine Eliezer means when he talks about “humanity growing up together”.
This growing-up process currently involves raising children, which can be viewed as a crude way of rewriting your personality from scratch, and excising vestiges of values you no longer endorse. It’s been an integral part of every culture’s moral evolution, and something like it needs to be part of CEV if it’s going to actually converge.
Maybe it’s true if you also specify “if they were fully capable of modifying their own moral intuitions.” I have an intuition (an unexamined belief? a hope? a sci-fi trope?) that humanity as a whole will continue to evolve morally and roughly converge on a morality that resembles current first-world liberal values more than, say, Old Testament values. That is, it would converge, in the limit of global prosperity and peace and dialogue, and assuming no singularity occurs and the average lifespan stays constant. You can call this naive if you want to; I don’t know whether it’s true. It’s what I imagine Eliezer means when he talks about “humanity growing up together”.
This growing-up process currently involves raising children, which can be viewed as a crude way of rewriting your personality from scratch, and excising vestiges of values you no longer endorse. It’s been an integral part of every culture’s moral evolution, and something like it needs to be part of CEV if it’s going to actually converge.