Here’s what would make me question Nate’s credibility: If we look at the 538′s former projections and compare them to the betting markets prior to election, make Kelly bets, and lose money.[1] Note that 538 can be on the wrong side of the market some of the time and still be profitable.
Saying Nate lacks enough credibility when he is well calibrated and on the right side of the market doesn’t make any sense and seems like an extreme case of hindsight bias. For example, you use the 2016 election to undermine 538′s credibility, but (if I recall correctly) 538′s forecast gave Trump a greater chance than the market’s odds implied, so if you bet based on 538′s forecast you would have bet on Trump and made money. I could be remembering wrong, so if someone links this data, they’d get an upvote from me.
I’m too lazy to look it up, but I did research this a couple of weeks ago and found that 538 had indeed outperformed the markets both in 2008 and 2016 (I wasn’t able to find data for 2012). This is not very informative, though, since it’s just a couple of cases. Much better is to look at the state-level predictions and use brier scores as a measure of forecasting performance.
And this year too. I agree about the small sample size if we’re just asking “Who will be president?” We should use every one of 538′s predictions that could have been used to make a bet in a liquid betting market.
If a profitable set of Kelly/Markowitz bets (under all election orderings) has a worse Brier score than the market’s Brier score, I’d be more interested in the profit of the bets, since we’re talking about betting money.
Here’s what would make me question Nate’s credibility: If we look at the 538′s former projections and compare them to the betting markets prior to election, make Kelly bets, and lose money.[1] Note that 538 can be on the wrong side of the market some of the time and still be profitable.
Saying Nate lacks enough credibility when he is well calibrated and on the right side of the market doesn’t make any sense and seems like an extreme case of hindsight bias. For example, you use the 2016 election to undermine 538′s credibility, but (if I recall correctly) 538′s forecast gave Trump a greater chance than the market’s odds implied, so if you bet based on 538′s forecast you would have bet on Trump and made money. I could be remembering wrong, so if someone links this data, they’d get an upvote from me.
Looking at all orderings of elections
I’m too lazy to look it up, but I did research this a couple of weeks ago and found that 538 had indeed outperformed the markets both in 2008 and 2016 (I wasn’t able to find data for 2012). This is not very informative, though, since it’s just a couple of cases. Much better is to look at the state-level predictions and use brier scores as a measure of forecasting performance.
538 totally outperformed in 2012 on intrade—it seems like there were whales pushing up the romney price on intrade.
And this year too. I agree about the small sample size if we’re just asking “Who will be president?” We should use every one of 538′s predictions that could have been used to make a bet in a liquid betting market.
If a profitable set of Kelly/Markowitz bets (under all election orderings) has a worse Brier score than the market’s Brier score, I’d be more interested in the profit of the bets, since we’re talking about betting money.