What was the result of the IARPA prediction contest (2010-2015)?
Below I present what seem to me very basic questions about the results. I have read vague statements about the results that sound like people are willing to answer these questions, but the details seem oddly elusive. Is there is some write-up I am missing?
How many teams were there? 5 academic teams? What were their names, schools, or PIs? What was the “control group”? Were there two, an official control group and another group consisting of intelligence analysts with access to classified information? Added: perhaps a third control group “a prediction market operating within the IARPA tournament.”
What were the Brier scores of the various teams in various years?
When Tetlock says that A did 10% better than B, does he mean that the Brier score of A was 90% of the Brier score of B?
I can identify 4 schools involved, composing 3-4 teams: GJP (Berkeley: Tetlock, Mellers, Moore) DAGGRE/SciCast (GMU: Twardy, Hanson) Michigan, MIT − 2 teams or a joint team?
Were there two, an official control group and another group consisting of intelligence analysts with access to classified information?
In Superforcasting Tetlock writes that the main documents of the comparison between the GJP forecasters against the intelligence analysts with access to classified information is classified. Tetlock doesn’t directly say something about that comparison but reports in his book that a newspaper article says that the GJP forecasters were 30% better (if I remember right).
Here is the leak. It says that the superforecasters averaged 30% better than the classified analysts. Presumably that’s the 2012-2013 season only and we won’t hear about other years.
What is weird is that other sources talk about “the control group” and for years I thought that this was the control group. But Tetlock implies that he doesn’t have access to the comparison with the classified group, but that he does have access to the comparison with the control group. In particular, he mentions that IARPA set a 4th year target of beating the control group by 50% and I think he says that he achieved that the first or second year. So that isn’t the classified comparison. I guess it is possible to reconcile the two comparisons by positing that the superforecasters were 30% better, but that GJP, after extremizing, was more than 50% better. But I think that there were two groups.
That’s what I thought when I saw the passage quoted from the book (p95), but then I got the book and looked at the endnote (p301) and Tetlock says:
I am willing to make a big reputational bet that the superforecasters beat the intelligence analysts in each year in which such comparisons were possible.
which must be illegal if he has seen the comparisons.
He likely worked with a censor about how and what he can write. I think that line can be very well explained as the result of a compromise with the censor.
What was the result of the IARPA prediction contest (2010-2015)?
Below I present what seem to me very basic questions about the results. I have read vague statements about the results that sound like people are willing to answer these questions, but the details seem oddly elusive. Is there is some write-up I am missing?
How many teams were there? 5 academic teams? What were their names, schools, or PIs? What was the “control group”? Were there two, an official control group and another group consisting of intelligence analysts with access to classified information?
Added: perhaps a third control group “a prediction market operating within the IARPA tournament.”
What were the Brier scores of the various teams in various years?
When Tetlock says that A did 10% better than B, does he mean that the Brier score of A was 90% of the Brier score of B?
I can identify 4 schools involved, composing 3-4 teams:
GJP (Berkeley: Tetlock, Mellers, Moore)
DAGGRE/SciCast (GMU: Twardy, Hanson)
Michigan, MIT − 2 teams or a joint team?
In Superforcasting Tetlock writes that the main documents of the comparison between the GJP forecasters against the intelligence analysts with access to classified information is classified. Tetlock doesn’t directly say something about that comparison but reports in his book that a newspaper article says that the GJP forecasters were 30% better (if I remember right).
Here is the leak. It says that the superforecasters averaged 30% better than the classified analysts. Presumably that’s the 2012-2013 season only and we won’t hear about other years.
What is weird is that other sources talk about “the control group” and for years I thought that this was the control group. But Tetlock implies that he doesn’t have access to the comparison with the classified group, but that he does have access to the comparison with the control group. In particular, he mentions that IARPA set a 4th year target of beating the control group by 50% and I think he says that he achieved that the first or second year. So that isn’t the classified comparison. I guess it is possible to reconcile the two comparisons by positing that the superforecasters were 30% better, but that GJP, after extremizing, was more than 50% better. But I think that there were two groups.
I’m not sure that X% better has a unit that’s always the same.
I don’t think that’s the case. It’s rather that it’s classified information that he can’t reveal directly because it’s classified.
That’s what I thought when I saw the passage quoted from the book (p95), but then I got the book and looked at the endnote (p301) and Tetlock says:
which must be illegal if he has seen the comparisons.
He likely worked with a censor about how and what he can write. I think that line can be very well explained as the result of a compromise with the censor.