I’ve heard some people try to claim silly things like, the probability that you’re telling the truth is counterbalanced by the probability that you’ll kill 3↑↑↑3 people instead, or something else with a conveniently equal and opposite utility. But there’s no way that things would balance out exactly in practice
With a human’s bounded computational resources, maybe assuming that it balances out is the best you can do. You have to make simplifying assumptions if you want to reason about numbers as large as 3↑↑↑3.
But we can see why the probability isn’t counterbalanced without having a visceral grasp on the quantities involved. There may be some uncertainty in our view that there aren’t enough counterbalancing forces we’ve taken into account, but in practice uncertainty almost always places you some nontrivial distance away from .5 credence. We still have to have credence levels, even about quantities that are very uncertain or beyond our ability to compute. Metauncertainty won’t drag your confidence to .5 unless your uncertainty disproportionately supports ‘I’m underestimating the likelihood that there are counterbalancing risks’ over ‘I’m overestimating the likelihood that there are counterbalancing risks’.
I think this comes down to bounded computation
With a human’s bounded computational resources, maybe assuming that it balances out is the best you can do. You have to make simplifying assumptions if you want to reason about numbers as large as 3↑↑↑3.
But we can see why the probability isn’t counterbalanced without having a visceral grasp on the quantities involved. There may be some uncertainty in our view that there aren’t enough counterbalancing forces we’ve taken into account, but in practice uncertainty almost always places you some nontrivial distance away from .5 credence. We still have to have credence levels, even about quantities that are very uncertain or beyond our ability to compute. Metauncertainty won’t drag your confidence to .5 unless your uncertainty disproportionately supports ‘I’m underestimating the likelihood that there are counterbalancing risks’ over ‘I’m overestimating the likelihood that there are counterbalancing risks’.