Someone is well-calibrated if the things they predict with X% chance of happening in fact occur X% of the time. Importantly, calibration is not the same as accuracy. Calibration is about accurately assessing how good your predictions are, not making good predictions. Person A, whose predictions are marginally better than chance (60% of them come true when choosing from two options) and who is precisely 60% confident in their choices, is perfectly calibrated. In contrast, Person B, who is 99% confident in their predictions, and right 90% of the time, is more accurate than Person A, but less well-calibrated.
See also: Betting, Epistemic Modesty, Forecasting & Prediction
Being well-calibrated has value for rationalists separately from accuracy. Among other things, being well-calibrated lets you make good bets / make good decisions, communicate information helpfully to others if they know you to be well-calibrated (See Group Rationality), and helps prioritize which information is worth acquiring.
Note that all expressions of quantified confidence in beliefs can be well- or poorly- calibrated. For example, calibration applies to whether a person’s 95% confidence intervals captures the true outcome 95% of the time.
List of Calibration Exercises
(From: https://www.lesswrong.com/posts/LdFbx9oqtKAAwtKF3/list-of-probability-calibration-exercises)
http://acritch.com/media/mphd/calibration-exercises.pdf
All different URLs for the same application:
https://play.google.com/store/apps/details?id=com.the_calibration_game
https://outsidetheasylum.blog/probability-calibration/
Exercises that are dead/unmaintained:
http://web.archive.org/web/20100529074053/http://www.acceleratingfuture.com/tom/?p=129
http://credencecalibration.com/ (dead link)
https://calibration.lazdini.lv/ (dead link)
http://web.archive.org/web/20161020032514/http://calibratedprobabilityassessment.org/
https://predictionbook.com/credence_games/try
Read-only; recommends https://fatebook.io/
https://calibration-training.netlify.app/ (dead link)
http://www.2pih.com/caltest/ (broken)
@Jim Fisher what’s your reasoning for removing the archive.org links?
I don’t think I removed any archive.org links. My intention was to remove dead links, and to make it clearer which links are worth visiting. Please revert if I’ve made a mistake or you disagree with my intention.