They may include values or desires we would consciously prefer to eradicate entirely, such as a drive for fighting or for making war. If you thought that 1) most people in history enjoyed and desired war, and 2) this was due to a feature of their builtin cognitive architecture that said “when in situation X, conquer your neighbors”—would you want to include this in the CEV?
CEV is supposed to incorporate not only the things you want (or enjoy), but also the things you want to want (or don’t want to enjoy, in this case).
As Vladimir Nesov said, there are builtin preferences (which CEV takes into account), and then there are our conscious desires or “state of knowledge about preference”. The two may be in conflict in some cases.
How do you know that CEV won’t include something that all the humans alive today, on the conscious level, would find hateful?
CEV is supposed to incorporate not only the things you want (or enjoy), but also the things you want to want (or don’t want to enjoy, in this case).
Supposed to based on what evidence?
As Vladimir Nesov said, there are builtin preferences (which CEV takes into account), and then there are our conscious desires or “state of knowledge about preference”. The two may be in conflict in some cases.
How do you know that CEV won’t include something that all the humans alive today, on the conscious level, would find hateful?