[LINK] “Prediction Audits” for Nate Silver, Dave Weigel
Nate Silver (the NYT quantitative political analyst) and Dave Weigel (the Slate columnist) have started a good tradition, listing their worst predictions of 2011. (Silver also listed his best.)
If any other pundits are doing the same, link them here.
Now that the election is over, I would like to see someone calculate Nate Silver’s Brier score (including his Senate predictions), and also the Brier score for anyone else who gave that many probabilistic predictions.
Update: Gwern is working on this with me. The result will be a CFAR blog post.
http://appliedrationality.org/2012/11/09/was-nate-silver-the-most-accurate-2012-election-pundit/
http://www.gwern.net/2012%20election%20predictions
When I go to that post on the CFAR site I get a popup window that says:
If I hit cancel it goes away and immediately returns. I’m using Firefox 16.0.2.
Looks like the whole blog part of the site is infected.
Edit: I can’t pinpoint where the infection is and I can’t duplicate it on another computer, but in case it helps, here are the URLs of the jar files it’s trying to run:
I’m also putting a link to this info in the “Contact the CFAR Staff” form on the site.
Thanks. We have someone looking at it now.
In case they haven’t found it yet, it looks like the code that sets the mischief in motion is the following:
(LW’s Markdown processor may add angle brackets around the URL that aren’t really there.)
Thanks; I believe it’s now fixed.
Why doesn’t Jackman get a Brier score? He claims it’s .00991: http://jackman.stanford.edu/blog/?p=2602
Jackman only just released his data now (after twittering with me, incidentally, I was able to explain why his R Brier score wasn’t matching his hand-calculated Brier score) because he forgot to send it to me last night; and I’m running on fumes—we started this project from scratch yesterday at 5PM and I’ve been working on it ever since. EDIT: Looks like all the kerfluffle of new Brier/RMSE scores prodded Sam Wang into releasing his precise predictions too! Neat. EDITEDIT: I’ve gotten Jackman’s data, incorporated it, discovered an error in my own data, differed with Jackman, learned he regarded 5 states as such a sure thing he didn’t include probabilities while I had simply put in
NAs, and now we’ve converged on his Brier score. Phew! His current Brier score is 0.009713686, a bit worse than Silver’s 0.009113725, and both seem to be outperformed by Drew Linzer’s 0.003843257. Wang seems to’ve released the data, but the CSV is unlabeled and I have no idea what half the columns mean...I’d also like to include a random-guesser equivalent for RMSE… Tomorrow.
A better Brier random guesser and its RMSE equivalent are now in the R doc and hopefully the blog post will be updated shortly.
We only included people whose Brier scores we could calculate ourselves. We plan to add Jackman when we get his data.
Apparently a team at Penn is doing this as well:
http://jackman.stanford.edu/blog/?p=2602
Is there a free / registration only prediction market game? I’m too poor to gamble real money, but I’d like to see something that will allow gambling for points or some such, and introduce it to my circle of friends. Something that allows a wide variety of categories of bet, with the ability to add your own well-quantified predictions.
I assume that you’ve heard of PredictionBook? That’s not scored, but it is a good system.
Thank you. I will check it out.
Weigel only lists his four worst.
Fixed the post- thanks!