(I’ll back off the Superman analogy; I think it’s disanalogous b/c of the discontinuity thing you point out.)
Yeah I like the analogue “some basketball players are NBA players.” It makes it sound totally unsurprising, which it is.
I don’t agree that Vox is right, because:
- I can’t find any evidence for the claim that forecasting ability is power-law distributed, and it’s not clear what that would mean with Brier scores (as Unnamed points out).
- Their use of the term “discovered.”
I don’t think I’m just quibbling over semantics; I definitely had the wrong idea about superforecasters prior to thinking it through, it seems like Vox might have it too, and I’m concerned others who read the article will get the wrong idea as well.
From participating on Metaculus I certainly don’t get the sense that there are people who make uncannily good predictions. If you compare the community prediction to the Metaculus prediction, it looks like there’s a 0.14 difference in average log score, which I guess means a combination of the best predictors tends to put e^(0.14) or 1.15 times as much probability on the correct answer as the time-weighted community median. (The postdiction is better, but I guess subject to overfitting?) That’s substantial, but presumably the combination of the best predictors is better than every individual predictor. The Metaculus prediction also seems to be doing a lot worse than the community prediction on recent questions, so I don’t know what to make of that. I suspect that, while some people are obviously better at forecasting than others, the word “superforecasters” has no content outside of “the best forecasters” and is just there to make the field of research sound more exciting.
(I’ll back off the Superman analogy; I think it’s disanalogous b/c of the discontinuity thing you point out.)
Yeah I like the analogue “some basketball players are NBA players.” It makes it sound totally unsurprising, which it is.
I don’t agree that Vox is right, because:
- I can’t find any evidence for the claim that forecasting ability is power-law distributed, and it’s not clear what that would mean with Brier scores (as Unnamed points out).
- Their use of the term “discovered.”
I don’t think I’m just quibbling over semantics; I definitely had the wrong idea about superforecasters prior to thinking it through, it seems like Vox might have it too, and I’m concerned others who read the article will get the wrong idea as well.
From participating on Metaculus I certainly don’t get the sense that there are people who make uncannily good predictions. If you compare the community prediction to the Metaculus prediction, it looks like there’s a 0.14 difference in average log score, which I guess means a combination of the best predictors tends to put e^(0.14) or 1.15 times as much probability on the correct answer as the time-weighted community median. (The postdiction is better, but I guess subject to overfitting?) That’s substantial, but presumably the combination of the best predictors is better than every individual predictor. The Metaculus prediction also seems to be doing a lot worse than the community prediction on recent questions, so I don’t know what to make of that. I suspect that, while some people are obviously better at forecasting than others, the word “superforecasters” has no content outside of “the best forecasters” and is just there to make the field of research sound more exciting.
Agreed. As I said, “it is unlikely that there is a sharp cutoff at 2%, there isn’t a discontinuity, and power law is probably the wrong term.”