I’d personally be more interested in asking someone for their 95% CI than their 68% CI, if I had to ask them for exactly one of the two. (Although it might again depend on what exactly I plain to do with this estimate.)
I’m usually much more interested in a 68% CI (or a 50% CI) than a 95% CI because:
People in general arent super calibrated, especially at the tails
You won’t find out for a while how good their intervals are anyway
What happens most often is usually the main interest. (Although in some scenarios the tails are all that matters, so again, depends on context—emphasis usually). I would like people to normalise narrower confidence intervals more.
(as you note) the tails are often dominated by model failure, so you’re asking a question less about their forecast, and more about their estimate of model failure. I want information about their model of the world rather than their beliefs about where their beliefs breakdown.
If you are new to continuous predictions then you should focus on the 50% Interval as it gives you most information about your calibration, If you are skilled and use for example a t-distribution then you have σ for the trunk and ν for the tail, even then few predictions should land in the tails, so most data should provide more information about how to adjust σ, than how to adjust ν
Hot take: I think the focus 95% is an artifact of us focusing on p<0.05 in frequentest statistics.
I agree identifying model failure is something people can be good at (although I find people often forget to consider it). Pricing it they are usually pretty bad at.
I’m usually much more interested in a 68% CI (or a 50% CI) than a 95% CI because:
People in general arent super calibrated, especially at the tails
You won’t find out for a while how good their intervals are anyway
What happens most often is usually the main interest. (Although in some scenarios the tails are all that matters, so again, depends on context—emphasis usually). I would like people to normalise narrower confidence intervals more.
(as you note) the tails are often dominated by model failure, so you’re asking a question less about their forecast, and more about their estimate of model failure. I want information about their model of the world rather than their beliefs about where their beliefs breakdown.
I agree with both points
If you are new to continuous predictions then you should focus on the 50% Interval as it gives you most information about your calibration, If you are skilled and use for example a t-distribution then you have σ for the trunk and ν for the tail, even then few predictions should land in the tails, so most data should provide more information about how to adjust σ, than how to adjust ν
Hot take: I think the focus 95% is an artifact of us focusing on p<0.05 in frequentest statistics.
I agree identifying model failure is something people can be good at (although I find people often forget to consider it). Pricing it they are usually pretty bad at.