Failing to assign the correct probability given your information is a failure both of accuracy and of calibration.
Suppose you take a test of many multiple choice questions (say, 5 choices), and for each question I elicit from you your probability of having the right answer. Accuracy is graded by your total score on the test. Calibration is graded by your log-score on the probabilities. Our lottery enthusiast might think they’re 50% likely to have the right answer even when they are picking randomly—and because of this they will have a lower log score than someone who correctly thinks they have a 1⁄5 chance. These two people may have the same scores on the test, but they will have different scores on their ability to assign probabilities.
I intend to update the article later to include log error. Thanks!
The lottery example though is a perfect reason to be careful about how much importance you place on calibration over accuracy.
Failing to assign the correct probability given your information is a failure both of accuracy and of calibration.
Suppose you take a test of many multiple choice questions (say, 5 choices), and for each question I elicit from you your probability of having the right answer. Accuracy is graded by your total score on the test. Calibration is graded by your log-score on the probabilities. Our lottery enthusiast might think they’re 50% likely to have the right answer even when they are picking randomly—and because of this they will have a lower log score than someone who correctly thinks they have a 1⁄5 chance. These two people may have the same scores on the test, but they will have different scores on their ability to assign probabilities.
I have updated my post to respond to your concerns, expanding on your lottery example in particular. Let me know if I’ve adequately addressed them.