I just mean that it’s objectively and clearly judgeable a wrong now that the year is out.
This is a rare and valuable property, as one will learn after reading through a few compilations of predictions for 2011.
Thanks for the clarification; my first reading was that you were holding people to the wrong standard. If you made 10 predictions at 70% each, and 6 of them come true, then you should be lauded rather than criticized. If all 10 of them come true (and appear to be causally independent of each other), then you should be criticized for underconfidence.
I’m guessing that many of your 80% and 90% predictions would be positively correlated with one or two major trends (e.g. there was no global economic meltdown in 2010), so it’s not quite that simple. Looks like pretty good calibration to me overall.
The 70-90% bracket for me is ~70 predictions judged; I don’t think most or even many of them are economic-related (except in a weak sense).
If you wanted to check my intuition, there doesn’t seem to be any easy way to filter my user page for just the judged ones within a probability range short of downloading and processing the HTML, but you could look through a few dozen or score of the recently judged predictions (http://predictionbook.com/predictions/judged) since I have looked at every prediction on the site and registered my own probabilities for essentially every prediction which isn’t either sports or highly personal (and even then I’ve frequently given it a shot anyway).
Thanks for the clarification; my first reading was that you were holding people to the wrong standard. If you made 10 predictions at 70% each, and 6 of them come true, then you should be lauded rather than criticized. If all 10 of them come true (and appear to be causally independent of each other), then you should be criticized for underconfidence.
/looks at http://predictionbook.com/users/gwern
: (
I’m guessing that many of your 80% and 90% predictions would be positively correlated with one or two major trends (e.g. there was no global economic meltdown in 2010), so it’s not quite that simple. Looks like pretty good calibration to me overall.
The 70-90% bracket for me is ~70 predictions judged; I don’t think most or even many of them are economic-related (except in a weak sense).
If you wanted to check my intuition, there doesn’t seem to be any easy way to filter my user page for just the judged ones within a probability range short of downloading and processing the HTML, but you could look through a few dozen or score of the recently judged predictions (http://predictionbook.com/predictions/judged) since I have looked at every prediction on the site and registered my own probabilities for essentially every prediction which isn’t either sports or highly personal (and even then I’ve frequently given it a shot anyway).