Motivated errors are much less legible to the person who is motivated. The on-purposes-ness of motivated cognition is quite different from the on-purposeness of deliberate choice and I think treating them as the same leads to important failures.
If someone consciously lies* to me, it’s generally because there is no part of them that thinks it was important enough to [edit: epistemically] cooperate with me. They specifically considered, with their System 2, and/or their entire subagent parliament, whether it was a good idea to lie to me, then they chose to do so. I have basically no further interest in attempting to cooperate with such a person unless they put a lot of work into repairing the damage.
When someone motivatedly rationalizes at the subconscious level, my sense of what’s happening is some combination of
a) no one is really in control and it makes more sense to model them as a random collection of atoms doing stuff than at the ‘agent’ abstraction. The random collection of atoms might respond to incentives, but naively applying incentives to them won’t necessarily work the way you want. (I mostly don’t think this is a useful frame here but noting it for completeness)
b) insofar as there is an agent there, it’s often the case that there are multiple subagents that are compartmentalized from each other. If they’ve made their way to the rationalsphere, read the sequences, etc, then it’s highly likely that at least one subagent highly endorses not being motivated. But that agent may not have conscious access to the fact that they are making the motivated error.
The priority, in my mind, for conversations among people striving to discuss beliefs empirically/rationally rather than as tribal affiliation or point scoring, should be to make sure the subagent that cares about truth remains in control. (Otherwise you’ve already failed, or dramatically increased the difficulty, of having a conversation that isn’t about tribal affiliation and point scoring)
[*minor point, but there’s a large class of lies like jokes and stuff than I’m not counting here]
Motivated errors are much less legible to the person who is motivated. The on-purposes-ness of motivated cognition is quite different from the on-purposeness of deliberate choice and I think treating them as the same leads to important failures.
If someone consciously lies* to me, it’s generally because there is no part of them that thinks it was important enough to [edit: epistemically] cooperate with me. They specifically considered, with their System 2, and/or their entire subagent parliament, whether it was a good idea to lie to me, then they chose to do so. I have basically no further interest in attempting to cooperate with such a person unless they put a lot of work into repairing the damage.
When someone motivatedly rationalizes at the subconscious level, my sense of what’s happening is some combination of
a) no one is really in control and it makes more sense to model them as a random collection of atoms doing stuff than at the ‘agent’ abstraction. The random collection of atoms might respond to incentives, but naively applying incentives to them won’t necessarily work the way you want. (I mostly don’t think this is a useful frame here but noting it for completeness)
b) insofar as there is an agent there, it’s often the case that there are multiple subagents that are compartmentalized from each other. If they’ve made their way to the rationalsphere, read the sequences, etc, then it’s highly likely that at least one subagent highly endorses not being motivated. But that agent may not have conscious access to the fact that they are making the motivated error.
The priority, in my mind, for conversations among people striving to discuss beliefs empirically/rationally rather than as tribal affiliation or point scoring, should be to make sure the subagent that cares about truth remains in control. (Otherwise you’ve already failed, or dramatically increased the difficulty, of having a conversation that isn’t about tribal affiliation and point scoring)
[*minor point, but there’s a large class of lies like jokes and stuff than I’m not counting here]
Would you place motivated errors, generally, in the same bucket as confirmation bias type thinking?