This post has its flaws, as has been pointed out, but to add the required nuance would make a book (or at least a LW sequence) out of what is currently a good and provocative post.
The nuances I think are vital:
The explicit consequence-representing/world-modeling parts of our minds exist alongside direct “function calls” like reflexes.
The decision-theory framework only imperfectly describes even that CR/WM part of our minds.
It’s quite possible (and indeed virtually certain) that changing the payoff matrix has been done by evolution in certain cases, in addition to implementing a SAMEL-friendly decision theory.
It is clear that SAMELs are not generally consciously accepted by average human beings as reasons for action, to virtually any extent. The SAMEL seems really strange at first, and is accepted as valid (let alone as pervasive) only after long reflection. It’s not that we just don’t have the understanding of what we do, it seems that we have a mental block against that part of the understanding.
Thanks for comment! I had actually struggled to keep it from ballooning, and ended up leaving off the part where I applied it to Haidt’s work, but then decided I wasn’t familiar enough with it to do it justice.
Re 3 and 4, I thought I made clear that SAMELs are not consciously recognized as such by humans, e.g. in the Drescher quote, and when I mentioned that careful self-reflection came late in evolutionary history. However, they do feel differently to act on.
This post has its flaws, as has been pointed out, but to add the required nuance would make a book (or at least a LW sequence) out of what is currently a good and provocative post.
The nuances I think are vital:
The explicit consequence-representing/world-modeling parts of our minds exist alongside direct “function calls” like reflexes.
The decision-theory framework only imperfectly describes even that CR/WM part of our minds.
It’s quite possible (and indeed virtually certain) that changing the payoff matrix has been done by evolution in certain cases, in addition to implementing a SAMEL-friendly decision theory.
It is clear that SAMELs are not generally consciously accepted by average human beings as reasons for action, to virtually any extent. The SAMEL seems really strange at first, and is accepted as valid (let alone as pervasive) only after long reflection. It’s not that we just don’t have the understanding of what we do, it seems that we have a mental block against that part of the understanding.
Thanks for comment! I had actually struggled to keep it from ballooning, and ended up leaving off the part where I applied it to Haidt’s work, but then decided I wasn’t familiar enough with it to do it justice.
Re 3 and 4, I thought I made clear that SAMELs are not consciously recognized as such by humans, e.g. in the Drescher quote, and when I mentioned that careful self-reflection came late in evolutionary history. However, they do feel differently to act on.