Indeed, the probability that we don’t know what’s going on is non-negligible. What I’m suggesting is that we don’t have to assign a non-negligible probability to the specific hypothesis “this mugger is speaking the literal truth”—instead we avoid overconfidence by trying to consider all of the hypotheses that might hide behind the general assertion “our grasp on this situation is much less than we think” and try to use broader reference classes to see what the outcomes of various strategies might be in those instances, using the strategy you outline for the lottery.
Instead of thinking of the proposition H = the mugger is honest, and trying to calculate E(U|AH)P(H) + E(U|A~H)P(~H) where A is an action such as handing over your wallet and U is utility, you consider the hypotheses you’re really applying, T, that your general theory about muggers is sufficient to understand the situation, and ~T, that you just don’t have a handle on the situation. Then instead of directly using the mugger’s stated utility for the value you try to appeal to a simpler and more general theory to find a value for E(U|A~T). The more general “an offer from a stranger” reference class should suffice; you buy only a tiny minority of the things that you’re offered. Beyond that you have the “don’t know” reference class, but that has to have expected zero utility.
This argument doesn’t apply to any possibility that you are in a position to properly think about. You are in a position to assess the probability that Miriam Achaba wishes to entrust you with 25M USD, so you’re best advised not to reach for the “other” column on that one.
Indeed, the probability that we don’t know what’s going on is non-negligible. What I’m suggesting is that we don’t have to assign a non-negligible probability to the specific hypothesis “this mugger is speaking the literal truth”—instead we avoid overconfidence by trying to consider all of the hypotheses that might hide behind the general assertion “our grasp on this situation is much less than we think” and try to use broader reference classes to see what the outcomes of various strategies might be in those instances, using the strategy you outline for the lottery.
Not to engage in needless turnabout, but how does that translate into math?
Instead of thinking of the proposition H = the mugger is honest, and trying to calculate E(U|AH)P(H) + E(U|A~H)P(~H) where A is an action such as handing over your wallet and U is utility, you consider the hypotheses you’re really applying, T, that your general theory about muggers is sufficient to understand the situation, and ~T, that you just don’t have a handle on the situation. Then instead of directly using the mugger’s stated utility for the value you try to appeal to a simpler and more general theory to find a value for E(U|A~T). The more general “an offer from a stranger” reference class should suffice; you buy only a tiny minority of the things that you’re offered. Beyond that you have the “don’t know” reference class, but that has to have expected zero utility.
This argument doesn’t apply to any possibility that you are in a position to properly think about. You are in a position to assess the probability that Miriam Achaba wishes to entrust you with 25M USD, so you’re best advised not to reach for the “other” column on that one.