The study linked to meant next to nothing in my eyes. It studied political predictions in an election year by political hacks on tv. 2007-2008. Guess what? IN an election cycle that liberals beat conservatives, the liberal predictions more often came true than conservative predictions.
Reminds me of the reported models of mortgage securities, created using data from boom times only.
Krugman was competing with a bunch of other political hacks and columnists. I doubt that accuracy is the highest motivation for any of them. The political hacks want to curry support, and the columnists want to be invited on tv and have their articles read. I’d put at least 3 motivations above accuracy for that crowd: manipulate attitudes, throw red meat to their natural markets, and entertain. It’s Dark Arts, all the way, all the time.
This objection is not entirely valid, at least when it comes to Krugman. Krugman scored 17⁄19 mainly on economic predictions, and one of the two he got wrong looks like a pro-Republican prediction.
From their executive summary:
According to our regression analysis, liberals are better predictors than conservatives—even when taking out the Presidential and Congressional election questions.
That’s a good point. I didn’t read the whole thing, as the basic premise seemed flawed. That does seem like real information about Krugman’s accuracy. I’d still wonder about the independence of the predictions, though. Did the paper authors address that issue at all? I saw noise about “statistical significance”, so I assumed not.
Are the specific predictions available online? It seemed like they had a large sample of predictions, so I doubt they were in the paper. This gives my estimate of Krugman a slight uptick, but without the actual predictions and results, this data can’t do much more for me.
That is awesome! They actually put out their data.
Pretty much, Krugman successfully predicted that the downturn would last a while (2-6,8,15), made some obvious statements (7,9,12,16,17,18), was questionably supported on one (11), was unfairly said to miss another (14), hit on a political prediction (10), and missed on another (13).
He was 50-50 or said nothing, except for successfully predicting that the downturn would take at least a couple of years, which wasn’t going out too far on a limb itself.
So I can’t say that I’m impressed much with the authors of the study, as their conclusions about Krugman seem like a gross distortion to me, but I am very impressed that they released their data. That was civilized.
Yes, and the paper had several other big problems. For example, it didn’t treat mild belief and certainty differently; someone who suspected Hilary might be the Democratic Nominee was treated as harshly as someone who was 100% sure the Danish were going to invade.
Worse, people get marked down for making conditional predictions whose antecedent was not satisfied! And then they have the audacity to claim that they’ve discovered that making conditional predictions predicts low accuracy.
They also penalise people for hedging, yet surely a hedged prediction is better than no prediction at all?
it didn’t treat mild belief and certainty differently;
It did. Per the paper, the confidences of the predictions were rated on a scale from 1 to 5, where 1 is “No chance of occurring” and 5 is “Definitely will occur”. They didn’t use this in their top-level rankings because they felt it was “accurate enough” without that, but they did use it in their regressions.
Worse, people get marked down for making conditional predictions whose antecedent was not satisfied!
They did not. Per the paper, those were simply thrown out (as people do on PredictionBook).
They also penalise people for hedging, yet surely a hedged prediction is better than no prediction at all?
I agree here, mostly. Looking through the predictions they’ve marked as hedging, some seem like sophistry but some seem like reasonable expressions of uncertainty; if they couldn’t figure out how to properly score them they should have just left them out.
If you think you can improve on their methodology, the full dataset is here: .xls.
Incidentally, the best way to make conditional predictions is to convert them to explicit disjunctions. For example, in November I wanted to predict that “If Mitt Romney loses the primary election, Barack Obama will win the general election.” This is actually logically equivalent to “Either Mitt Romney or Barack Obama will win the 2012 Presidential Election,” barring some very unlikely events, so I posted that instead, and so I won’t have to withdraw the prediction when Romney wins the primary.
While that may be best with current PB, I think conditional predictions are useful.
If you are only interested in truth values and not the strength of the prediction, then it is logically equivalent, but the number of points you get is not the same. The purpose of a conditional probability is to take a conditional risk. If Romney is nominated, you get a gratuitous point for this prediction. Of course, simply counting predictions is easy to game, which is why we like to indicate the strength of the prediction, as you do with this one on PB. But turning a conditional prediction into an absolute prediction changes its probability and thus its effect on your calibration score. To a certain extent, it amounts to double counting the prediction about the hypothesis.
The first version doesn’t have that part either- he’s predicting that if Romney gets eliminated in the primaries, ie Gingrich, Santorum, or Paul is the Republican nominee, then Obama will win.
it didn’t treat mild belief and certainty differently;
… they did use it in their regressions.
Sure, so we learn about how confidence is correlated with binary accuracy. But they don’t take into account that being very confident and wrong should be penalised more than being slightly confident and wrong.
Worse, people get marked down for making conditional predictions whose antecedent was not satisfied! And then they have the audacity to claim that they’ve discovered that making conditional predictions predicts low accuracy.
Why do you think this? Doesn’t seem true at all to me.
Looking at the spreadsheet there are many judgements left blank with the phrase “conditional not met.” They are not counted in the total number of predictions.
OK, I don’t want to get off-topic. EY doesn’t practice the Dark Arts (at least, I hope not).
A lot of what EY writes makes sense to me. And I’d like to believe that we’ll be sipping champagne on the other side of the galaxy when the last star in the Milky Way burns out (and note that I’m not saying that he’s predicting that will happen). But I’m not a physicist or AI researcher—I want some way to know how much to trust what he writes. Is anything that he’s said or done falsifiable? Has he ever publicly made his beliefs pay rent? I want to believe in a friendly AI future… but I’m not going to believe for the sake of believing.
The study linked to meant next to nothing in my eyes. It studied political predictions in an election year by political hacks on tv. 2007-2008. Guess what? IN an election cycle that liberals beat conservatives, the liberal predictions more often came true than conservative predictions.
Reminds me of the reported models of mortgage securities, created using data from boom times only.
Krugman was competing with a bunch of other political hacks and columnists. I doubt that accuracy is the highest motivation for any of them. The political hacks want to curry support, and the columnists want to be invited on tv and have their articles read. I’d put at least 3 motivations above accuracy for that crowd: manipulate attitudes, throw red meat to their natural markets, and entertain. It’s Dark Arts, all the way, all the time.
This objection is not entirely valid, at least when it comes to Krugman. Krugman scored 17⁄19 mainly on economic predictions, and one of the two he got wrong looks like a pro-Republican prediction.
From their executive summary:
From the paper:
That’s a good point. I didn’t read the whole thing, as the basic premise seemed flawed. That does seem like real information about Krugman’s accuracy. I’d still wonder about the independence of the predictions, though. Did the paper authors address that issue at all? I saw noise about “statistical significance”, so I assumed not.
Are the specific predictions available online? It seemed like they had a large sample of predictions, so I doubt they were in the paper. This gives my estimate of Krugman a slight uptick, but without the actual predictions and results, this data can’t do much more for me.
Here’s their xls with the predictions
That is awesome! They actually put out their data.
Pretty much, Krugman successfully predicted that the downturn would last a while (2-6,8,15), made some obvious statements (7,9,12,16,17,18), was questionably supported on one (11), was unfairly said to miss another (14), hit on a political prediction (10), and missed on another (13).
He was 50-50 or said nothing, except for successfully predicting that the downturn would take at least a couple of years, which wasn’t going out too far on a limb itself.
So I can’t say that I’m impressed much with the authors of the study, as their conclusions about Krugman seem like a gross distortion to me, but I am very impressed that they released their data. That was civilized.
Yes, and the paper had several other big problems. For example, it didn’t treat mild belief and certainty differently; someone who suspected Hilary might be the Democratic Nominee was treated as harshly as someone who was 100% sure the Danish were going to invade.
Worse, people get marked down for making conditional predictions whose antecedent was not satisfied! And then they have the audacity to claim that they’ve discovered that making conditional predictions predicts low accuracy.
They also penalise people for hedging, yet surely a hedged prediction is better than no prediction at all?
It did. Per the paper, the confidences of the predictions were rated on a scale from 1 to 5, where 1 is “No chance of occurring” and 5 is “Definitely will occur”. They didn’t use this in their top-level rankings because they felt it was “accurate enough” without that, but they did use it in their regressions.
They did not. Per the paper, those were simply thrown out (as people do on PredictionBook).
I agree here, mostly. Looking through the predictions they’ve marked as hedging, some seem like sophistry but some seem like reasonable expressions of uncertainty; if they couldn’t figure out how to properly score them they should have just left them out.
If you think you can improve on their methodology, the full dataset is here: .xls.
Incidentally, the best way to make conditional predictions is to convert them to explicit disjunctions. For example, in November I wanted to predict that “If Mitt Romney loses the primary election, Barack Obama will win the general election.” This is actually logically equivalent to “Either Mitt Romney or Barack Obama will win the 2012 Presidential Election,” barring some very unlikely events, so I posted that instead, and so I won’t have to withdraw the prediction when Romney wins the primary.
While that may be best with current PB, I think conditional predictions are useful.
If you are only interested in truth values and not the strength of the prediction, then it is logically equivalent, but the number of points you get is not the same. The purpose of a conditional probability is to take a conditional risk. If Romney is nominated, you get a gratuitous point for this prediction. Of course, simply counting predictions is easy to game, which is why we like to indicate the strength of the prediction, as you do with this one on PB. But turning a conditional prediction into an absolute prediction changes its probability and thus its effect on your calibration score. To a certain extent, it amounts to double counting the prediction about the hypothesis.
This is less specific than the first prediction. The second version loses the part where you predict obama will beat romney
The first version doesn’t have that part either- he’s predicting that if Romney gets eliminated in the primaries, ie Gingrich, Santorum, or Paul is the Republican nominee, then Obama will win.
you’re right, I misread.
Sure, so we learn about how confidence is correlated with binary accuracy. But they don’t take into account that being very confident and wrong should be penalised more than being slightly confident and wrong.
I misread; you are right
That made me giggle.
Why do you think this? Doesn’t seem true at all to me. Looking at the spreadsheet there are many judgements left blank with the phrase “conditional not met.” They are not counted in the total number of predictions.
OK, I don’t want to get off-topic. EY doesn’t practice the Dark Arts (at least, I hope not).
A lot of what EY writes makes sense to me. And I’d like to believe that we’ll be sipping champagne on the other side of the galaxy when the last star in the Milky Way burns out (and note that I’m not saying that he’s predicting that will happen). But I’m not a physicist or AI researcher—I want some way to know how much to trust what he writes. Is anything that he’s said or done falsifiable? Has he ever publicly made his beliefs pay rent? I want to believe in a friendly AI future… but I’m not going to believe for the sake of believing.
Previous discussion of Krugman’s accuracy here.