In other words, the wrong way to determine your utility function is to think about what decisions you have made, or feel that you would make, in different situations.
Not quite. Examining in detail the actual reasons for your decisions made in the past or those you could’ve counterfactually made can show you which real-world factor or cognitive mechanism should’ve been different how, which would allow you to obtain a specific better judgment in that situation (or a more general class of situations). This is the way you can point out cognitive biases. It’s wrong to discard your existing intuitions, even if you know that they are no good in the greater scheme of things, because they may well be the best you’ve got, or the best raw material you’ve got for developing better ones.
“I’m always timid in social situations therefore I want to be timid in social situations” is invalid reasoning.
To a first approximation, and as a heuristic rule, it is valid. There are specific additional reasons to believe the conclusion invalid, and they have to do with things other than the way the initial faulty conclusion was generated. You believe the reasoning invalid because you know the conclusion to be invalid, but not the other way around.
Not quite. Examining in detail the actual reasons for your decisions made in the past or those you could’ve counterfactually made can show you which real-world factor or cognitive mechanism should’ve been different how, which would allow you to obtain a specific better judgment in that situation (or a more general class of situations). This is the way you can point out cognitive biases. It’s wrong to discard your existing intuitions, even if you know that they are no good in the greater scheme of things, because they may well be the best you’ve got, or the best raw material you’ve got for developing better ones.
Agreed—I was too dismissive of what can be learnt from past decision-making experiences.
I was just pointing out that “I’m always timid in social situations therefore I want to be timid in social situations” is invalid reasoning.
To a first approximation, and as a heuristic rule, it is valid. There are specific additional reasons to believe the conclusion invalid, and they have to do with things other than the way the initial faulty conclusion was generated. You believe the reasoning invalid because you know the conclusion to be invalid, but not the other way around.
I meant “want” as in “this is one of my life goals; I would not wish to self-modify to be any other way”