(If this makes no sense, then ignore it): Using an arbitrary distribution for predictions, then use its CDF (Universality of the Uniform) to convert to U(0,1), and then transform to z-score using the inverse CDF (percentile point function) of the Unit Normal. Finally use this as zi in when calculating your calibration.
This glosses over an important issue: namely, how do you find (and score once you do find it) which distribution is correct?
(This issue occurs surprisingly often in my experience—one person thinks that a distribution has a thin tail, and another thinks that the distribution has a fat tail, but there’s little enough data that both estimates are plausible. Of course, they give very different predictions for long-tail events.)
(Also, P(0 votes for Biden) is arguably significantly more likely than P(1 vote for Biden), in which case even the example chosen suffers from this issue of choosing the wrong distribution.)
I agree tails are important, but for callibration few of your predictions should land in the tail, so imo you should focus on getting the trunk of the distribution right first, and the later learn to do overdispersed predictions, there is no closed form solution to callibration for a t distribution, but there is for a normal, so for pedagogical reasons I am biting the bullet and asuming the normal is correct :), part 10 in this series 3 years in the future may be some black magic of the posterior of your t predictions using HMC to approximate the 2d posterior of sigma and nu ;), and then you can complain “but what about skewed distributios” :P
This glosses over an important issue: namely, how do you find (and score once you do find it) which distribution is correct?
(This issue occurs surprisingly often in my experience—one person thinks that a distribution has a thin tail, and another thinks that the distribution has a fat tail, but there’s little enough data that both estimates are plausible. Of course, they give very different predictions for long-tail events.)
(Also, P(0 votes for Biden) is arguably significantly more likely than P(1 vote for Biden), in which case even the example chosen suffers from this issue of choosing the wrong distribution.)
I agree tails are important, but for callibration few of your predictions should land in the tail, so imo you should focus on getting the trunk of the distribution right first, and the later learn to do overdispersed predictions, there is no closed form solution to callibration for a t distribution, but there is for a normal, so for pedagogical reasons I am biting the bullet and asuming the normal is correct :), part 10 in this series 3 years in the future may be some black magic of the posterior of your t predictions using HMC to approximate the 2d posterior of sigma and nu ;), and then you can complain “but what about skewed distributios” :P
...which then systematically underestimates long-tail risk, often significantly.