I may well be missing something, but, like Elf, I don’t see how Eliezer’s “evolutionary-cognitive boundary” can be well-defined.
If humans were ideal optimizers, a sharp distinction would make sense. Humans would come with genetically encoded objective functions. Some sub-goals (e.g., “keep your kids alive”) would be encoded in our objective functions, and would be goals we intrinsically and permanently cared about. Other sub-goals (e.g., “wear your safety belt”) would be consciously computed heuristics that we had no intrinsic attachment to, and that we would effortlessly lose when new information became available. “Beliefs” would be produced by a blend of evolutionary and individual computation (we would have genetically encoded heuristics for using sense-data and neurons to form accurate predictions, as in human language-learning), while “preference” would have a sharp line division.
But humans aren’t ideal optimizers; humans are a kludged-together mess of preferences, habits, reflexes, context-sensitive heuristics, conscious and non-conscious “beliefs”, and faculties for updating the same from experience. Our “intrinsic preferences” change in a manner not unlike the manner our beliefs update (based, again, on evolutionarily created algorithms that make use of sense-data and neurology; e.g., heuristics for updating the “intrinsic” attractiveness of your mate based on others’ views of his attractiveness). Even at a single time, behavior can’t always be parsed into (temporary) “beliefs” and (temporary) “preferences”; sometimes our apparent “preferences” vary so dramatically depending on context or framing that I’m not sure we should be regarded as having “preferences” at all (think Cialdini), and sometimes we just have a mix of habits, tendencies, emotions, and actions that acts vaguely purposive.
For example:
*Fear of heights. Humans are designed (based on evolutionary computation) to acquire such fears easily; still, my fear is partially (though not fully) responsive to my conscious beliefs about how dangerous a particular cliff is, and is also partially (though not fully) responsive to subconscious computations that update from whether I’ve been badly hurt falling from a height (even if I consciously know that this height is different and harmless). Should my fear here be regarded as an intrinsic preference, a conscious subgoal of “avoid harm”, an evolutionarily encoded “belief” (that is felt as an emotion, but updated from new data), or what?
Can someone who thinks a sharp line distinction is desirable explain how to draw that line for fear of heights or a similar example? Am I under-estimating the extent to which humans can be thought of as “having intrinsic preferences”?
I may well be missing something, but, like Elf, I don’t see how Eliezer’s “evolutionary-cognitive boundary” can be well-defined.
If humans were ideal optimizers, a sharp distinction would make sense. Humans would come with genetically encoded objective functions. Some sub-goals (e.g., “keep your kids alive”) would be encoded in our objective functions, and would be goals we intrinsically and permanently cared about. Other sub-goals (e.g., “wear your safety belt”) would be consciously computed heuristics that we had no intrinsic attachment to, and that we would effortlessly lose when new information became available. “Beliefs” would be produced by a blend of evolutionary and individual computation (we would have genetically encoded heuristics for using sense-data and neurons to form accurate predictions, as in human language-learning), while “preference” would have a sharp line division.
But humans aren’t ideal optimizers; humans are a kludged-together mess of preferences, habits, reflexes, context-sensitive heuristics, conscious and non-conscious “beliefs”, and faculties for updating the same from experience. Our “intrinsic preferences” change in a manner not unlike the manner our beliefs update (based, again, on evolutionarily created algorithms that make use of sense-data and neurology; e.g., heuristics for updating the “intrinsic” attractiveness of your mate based on others’ views of his attractiveness). Even at a single time, behavior can’t always be parsed into (temporary) “beliefs” and (temporary) “preferences”; sometimes our apparent “preferences” vary so dramatically depending on context or framing that I’m not sure we should be regarded as having “preferences” at all (think Cialdini), and sometimes we just have a mix of habits, tendencies, emotions, and actions that acts vaguely purposive.
For example: *Fear of heights. Humans are designed (based on evolutionary computation) to acquire such fears easily; still, my fear is partially (though not fully) responsive to my conscious beliefs about how dangerous a particular cliff is, and is also partially (though not fully) responsive to subconscious computations that update from whether I’ve been badly hurt falling from a height (even if I consciously know that this height is different and harmless). Should my fear here be regarded as an intrinsic preference, a conscious subgoal of “avoid harm”, an evolutionarily encoded “belief” (that is felt as an emotion, but updated from new data), or what?
Can someone who thinks a sharp line distinction is desirable explain how to draw that line for fear of heights or a similar example? Am I under-estimating the extent to which humans can be thought of as “having intrinsic preferences”?