What do you believe about these immutable, universal preferences?
Here are some potential problems I see with these theorized builtin preferences, since we don’t know what they actually are yet:
They may conflict with our consciously held morals or desires: e.g., they may not include compassion or altruism for anyone we never met face to face. They may even conflict with both our own morals and with Gork’s morals, at the same time. In that case, why shouldn’t we privilege our conscious desires?
They may not be very interesting: just “want to have food and comfort, sex, social status, children”. They wouldn’t include many things we consciously want because those thing evolved out of subverted button-pushing or as hyperstimuli—such as scientific research. Why should we choose to discard such values just because they aren’t embodied in our hardware?
They may be susceptible to many cognitive traps and dead-ends (e.g. wireheading) that we can only work around using conscious thought and our consciously held values.
They may include values or desires we would consciously prefer to eradicate entirely, such as a drive for fighting or for making war. If you thought that 1) most people in history enjoyed and desired war, and 2) this was due to a feature of their builtin cognitive architecture that said “when in situation X, conquer your neighbors”—would you want to include this in the CEV?
They may include features that optimize inclusive genetic fitness at the expense of the user, e.g. causing emotional suffering as negative feedback.
They may include values or desires we would consciously prefer to eradicate entirely, such as a drive for fighting or for making war. If you thought that 1) most people in history enjoyed and desired war, and 2) this was due to a feature of their builtin cognitive architecture that said “when in situation X, conquer your neighbors”—would you want to include this in the CEV?
CEV is supposed to incorporate not only the things you want (or enjoy), but also the things you want to want (or don’t want to enjoy, in this case).
As Vladimir Nesov said, there are builtin preferences (which CEV takes into account), and then there are our conscious desires or “state of knowledge about preference”. The two may be in conflict in some cases.
How do you know that CEV won’t include something that all the humans alive today, on the conscious level, would find hateful?
What do you believe about these immutable, universal preferences?
Here are some potential problems I see with these theorized builtin preferences, since we don’t know what they actually are yet:
They may conflict with our consciously held morals or desires: e.g., they may not include compassion or altruism for anyone we never met face to face. They may even conflict with both our own morals and with Gork’s morals, at the same time. In that case, why shouldn’t we privilege our conscious desires?
They may not be very interesting: just “want to have food and comfort, sex, social status, children”. They wouldn’t include many things we consciously want because those thing evolved out of subverted button-pushing or as hyperstimuli—such as scientific research. Why should we choose to discard such values just because they aren’t embodied in our hardware?
They may be susceptible to many cognitive traps and dead-ends (e.g. wireheading) that we can only work around using conscious thought and our consciously held values.
They may include values or desires we would consciously prefer to eradicate entirely, such as a drive for fighting or for making war. If you thought that 1) most people in history enjoyed and desired war, and 2) this was due to a feature of their builtin cognitive architecture that said “when in situation X, conquer your neighbors”—would you want to include this in the CEV?
They may include features that optimize inclusive genetic fitness at the expense of the user, e.g. causing emotional suffering as negative feedback.
CEV is supposed to incorporate not only the things you want (or enjoy), but also the things you want to want (or don’t want to enjoy, in this case).
Supposed to based on what evidence?
As Vladimir Nesov said, there are builtin preferences (which CEV takes into account), and then there are our conscious desires or “state of knowledge about preference”. The two may be in conflict in some cases.
How do you know that CEV won’t include something that all the humans alive today, on the conscious level, would find hateful?