It seems like you’re framing this in terms of “extreme probabilities are unlikely to be accurate”, but...
You give an example of 80% probabilities being inaccurate.
You use AGI risk as an example, which around here I often see estimates like “50% by this date, 75% by this date” and I get the impression you meant it to apply to that sort of thing too.
You can always make an extreme probability less extreme. Silly example: “99% chance of AGI tomorrow” becomes “49.5% chance of AGI tomorrow and I get heads on this coin toss”.
I feel like this kind of thing needs to be about inputs, not outputs. “If you find yourself calculating a probability under these circumstances, be suspicious”, not “if you find you calculated a probability of this level, be suspicious”.
Also… it seems like you’re assuming this as background:
Most of us are very much uncalibrated. We give 80%+ chance of success/completion to the next project even after failing a bunch of similar ones before. Even those of us who are well calibrated are still bad at the margins, where the probability is low (or, equivalently, high). Events we give the odds of 1% to can happen with the frequency of 20%.
And the rest of the post riffs off that. (Like, your examples seem like not “here are examples to convince you you’re uncalibrated” but “here are examples of how to deal with the fact that you’re uncalibrated” or something.)
But, citation needed.
I’ll grant the “most of us”. I recall the studies mentioned in HPMOR, along the lines of “you ask people when they’re 95% likely to finish only like a quarter finish by then. And you ask when they’re 50% likely to finish and it’s statistically indistinguishable”. I think to (reliably, reproducibly, robustly) get results like that, most of the people in those studies need to be poorly calibrated on the questions they’re being asked.
But the “even those of us”? Given that the first two words of the post are “yes, you”—that is, you project extreme confidence that this applies to the reader… how do you know that the reader is bad at the margins even if they’re well calibrated elsewhere?
Is this also true of superforecasters? Is it true of the sorts of people who say “man, I really don’t know. I think I’d be comfortable buying an implied probability of 0.01% and selling an implied probability of 1%, I know that’s a crazy big range but that’s where I am”?
(This seems like the sort of extreme confidence that you warn about in this very post. I know you admit to being more fallible than you think, but...)
I agree that there are people who don’t need this warning most of the time. Because they already double and triple check their estimates and are the first ones to admit to their fallibility. “Most of us” are habitually overconfident though. I also agree that the circumstances matter a lot, and some people in some circumstances can be accurate at 1% level, but most people in most circumstances aren’t. I’m guessing that superforecasters would not even try to estimate anything at 1% level, realizing they cannot do it well enough. We are most fallible when we don’t even realize we are calculating odds (there is a suitable HPMOR quote about that, too). Your example of giving a confidence interval or a range of probabilities is definitely an improvement over the usual Bayesian point estimates, but I don’t see any easily accessible version of the Bayes formula for ranges, though admittedly I’m not looking hard enough. In general, thinking in terms of distributions, not point estimates, seems like it would be progress. Mathematicians and physicists do that already in a professional setting.
It seems like you’re framing this in terms of “extreme probabilities are unlikely to be accurate”, but...
You give an example of 80% probabilities being inaccurate.
You use AGI risk as an example, which around here I often see estimates like “50% by this date, 75% by this date” and I get the impression you meant it to apply to that sort of thing too.
You can always make an extreme probability less extreme. Silly example: “99% chance of AGI tomorrow” becomes “49.5% chance of AGI tomorrow and I get heads on this coin toss”.
I feel like this kind of thing needs to be about inputs, not outputs. “If you find yourself calculating a probability under these circumstances, be suspicious”, not “if you find you calculated a probability of this level, be suspicious”.
Also… it seems like you’re assuming this as background:
And the rest of the post riffs off that. (Like, your examples seem like not “here are examples to convince you you’re uncalibrated” but “here are examples of how to deal with the fact that you’re uncalibrated” or something.)
But, citation needed.
I’ll grant the “most of us”. I recall the studies mentioned in HPMOR, along the lines of “you ask people when they’re 95% likely to finish only like a quarter finish by then. And you ask when they’re 50% likely to finish and it’s statistically indistinguishable”. I think to (reliably, reproducibly, robustly) get results like that, most of the people in those studies need to be poorly calibrated on the questions they’re being asked.
But the “even those of us”? Given that the first two words of the post are “yes, you”—that is, you project extreme confidence that this applies to the reader… how do you know that the reader is bad at the margins even if they’re well calibrated elsewhere?
Is this also true of superforecasters? Is it true of the sorts of people who say “man, I really don’t know. I think I’d be comfortable buying an implied probability of 0.01% and selling an implied probability of 1%, I know that’s a crazy big range but that’s where I am”?
(This seems like the sort of extreme confidence that you warn about in this very post. I know you admit to being more fallible than you think, but...)
I agree that there are people who don’t need this warning most of the time. Because they already double and triple check their estimates and are the first ones to admit to their fallibility. “Most of us” are habitually overconfident though. I also agree that the circumstances matter a lot, and some people in some circumstances can be accurate at 1% level, but most people in most circumstances aren’t. I’m guessing that superforecasters would not even try to estimate anything at 1% level, realizing they cannot do it well enough. We are most fallible when we don’t even realize we are calculating odds (there is a suitable HPMOR quote about that, too). Your example of giving a confidence interval or a range of probabilities is definitely an improvement over the usual Bayesian point estimates, but I don’t see any easily accessible version of the Bayes formula for ranges, though admittedly I’m not looking hard enough. In general, thinking in terms of distributions, not point estimates, seems like it would be progress. Mathematicians and physicists do that already in a professional setting.