Here is the leak. It says that the superforecasters averaged 30% better than the classified analysts. Presumably that’s the 2012-2013 season only and we won’t hear about other years.
What is weird is that other sources talk about “the control group” and for years I thought that this was the control group. But Tetlock implies that he doesn’t have access to the comparison with the classified group, but that he does have access to the comparison with the control group. In particular, he mentions that IARPA set a 4th year target of beating the control group by 50% and I think he says that he achieved that the first or second year. So that isn’t the classified comparison. I guess it is possible to reconcile the two comparisons by positing that the superforecasters were 30% better, but that GJP, after extremizing, was more than 50% better. But I think that there were two groups.
That’s what I thought when I saw the passage quoted from the book (p95), but then I got the book and looked at the endnote (p301) and Tetlock says:
I am willing to make a big reputational bet that the superforecasters beat the intelligence analysts in each year in which such comparisons were possible.
which must be illegal if he has seen the comparisons.
He likely worked with a censor about how and what he can write. I think that line can be very well explained as the result of a compromise with the censor.
Here is the leak. It says that the superforecasters averaged 30% better than the classified analysts. Presumably that’s the 2012-2013 season only and we won’t hear about other years.
What is weird is that other sources talk about “the control group” and for years I thought that this was the control group. But Tetlock implies that he doesn’t have access to the comparison with the classified group, but that he does have access to the comparison with the control group. In particular, he mentions that IARPA set a 4th year target of beating the control group by 50% and I think he says that he achieved that the first or second year. So that isn’t the classified comparison. I guess it is possible to reconcile the two comparisons by positing that the superforecasters were 30% better, but that GJP, after extremizing, was more than 50% better. But I think that there were two groups.
I’m not sure that X% better has a unit that’s always the same.
I don’t think that’s the case. It’s rather that it’s classified information that he can’t reveal directly because it’s classified.
That’s what I thought when I saw the passage quoted from the book (p95), but then I got the book and looked at the endnote (p301) and Tetlock says:
which must be illegal if he has seen the comparisons.
He likely worked with a censor about how and what he can write. I think that line can be very well explained as the result of a compromise with the censor.