(Upon further-further thought, is it something like the following internal dialogue? “I care about people close to me. I also care about the fate of mankind. I know that the fate of mankind as a whole is far more important than the fate of the people close to me. Since I value internal consistency, in order for my caring-mechanism to be consistent, my care for the fate of mankind must be proportional to my care for the people close to me. Since my caring mechanism is incapable of actually computing such a proportionality, the next best thing is to be consciously aware of how much it should care if it were able, and act accordingly.”)
I care about self-consistency, but being self-consistent is something that must happen naturally; I can’t self-consistently say “This feeling is self-inconsistent, therefore I will change this feeling to be self-consistent”
I actually think that your internal dialogue was a pretty accurate representation of what I was failing to say. And as for self consistency having to be natural, I agree, but if you’re aware that you’re being inconsistent, you can still alter your actions to try and correct for that fact.
I care about self-consistency, but being self-consistent is something that must happen naturally; I can’t self-consistently say “This feeling is self-inconsistent, therefore I will change this feeling to be self-consistent”
… Oh.
Hm. In that case, I think I’m still missing something fundamental.
I care about self-consistency because an inconsistent self is very strong evidence that I’m doing something wrong.
It’s not very likely that if I take the minimum steps to make the evidence of the error go away, I will make the error go away.
The general case of “find a self-inconsistency, make the minimum change to remove it” is not error-correcting.
I actually think that your internal dialogue was a pretty accurate representation of what I was failing to say. And as for self consistency having to be natural, I agree, but if you’re aware that you’re being inconsistent, you can still alter your actions to try and correct for that fact.