I recognize a very similar failure mode of instrumental rationality: I sometimes include in the decision process for an action not just the utility of that action itself, but also its probability. That is, I act on the expected utility of the action, not on its utility. Example:
I should hurry up enough to catch my train (hurrying up enough has high utility)
Based on experience, I probably won’t hurry up enough (hurrying up enough has low probability)
So the expected utility (utility*probability) of hurrying up enough is not very high
So I don’t hurry up enough
So I miss my train.
The mistake is to pay any attention to the expected utility (utility*probability) of an action, rather than just to its utility. The probability of what I will do is irrelevant to what I should do. The probability of an action should be the output, never the input of my decision. If one action has the highest utility, it should go to 100% probability (that is, I should do it) and all the alternative actions should go to 0 probability.
The scary thing is that recognizing this mistake doesn’t help with avoiding it.
I recognize a very similar failure mode of instrumental rationality: I sometimes include in the decision process for an action not just the utility of that action itself, but also its probability. That is, I act on the expected utility of the action, not on its utility. Example:
I should hurry up enough to catch my train (hurrying up enough has high utility)
Based on experience, I probably won’t hurry up enough (hurrying up enough has low probability)
So the expected utility (utility*probability) of hurrying up enough is not very high
So I don’t hurry up enough
So I miss my train.
The mistake is to pay any attention to the expected utility (utility*probability) of an action, rather than just to its utility. The probability of what I will do is irrelevant to what I should do. The probability of an action should be the output, never the input of my decision. If one action has the highest utility, it should go to 100% probability (that is, I should do it) and all the alternative actions should go to 0 probability.
The scary thing is that recognizing this mistake doesn’t help with avoiding it.