I think one really important decision-relevant question is:
“Do we need to have forecasters spend years forecasting questions before we can get a good sense of how good they are, or can we get most of that information with a quick (<1 week) test?”
My impression is that the Good Judgement Project used several tests to attempt to identify forecasters, but the tests didn’t predict the superforecasters as well as what some may have desired.
Do you think that almost all of this can be explained either by:
Diligence to the questions, similar to your example of the MMORPG?
Other simple things that we may be able to figure out in the next few years?
If so, I imagine the value of being a “superforecaster” would go down a bit, but the value of being “a superforecaster in expectation” would go up.
Yes—I suspect a large amount of the variance is explained by features we can measure, and the residual may be currently unexplained, but filtering on the features you can measure probably gets most of what is needed.
However, I don’t think the conclusion necessarily follows.
The problem is a causal reasoning / incentive issue (because of reasons) - just because people who update frequently do well doesn’t mean that telling people you’ll pay those who update frequently will cause them to do better now that they update more often. For instance, if you took MMORPG players and gave them money on condition that they spend money on the game, you’ll screw up the relationship between spending and success.
Fair point. I’m sure you expect some correlation between the use of reasonable incentive structures and useful updating though. It may not be perfect, but I’d be surprised if it were 0.
I think one really important decision-relevant question is:
My impression is that the Good Judgement Project used several tests to attempt to identify forecasters, but the tests didn’t predict the superforecasters as well as what some may have desired.
Do you think that almost all of this can be explained either by:
Diligence to the questions, similar to your example of the MMORPG?
Other simple things that we may be able to figure out in the next few years?
If so, I imagine the value of being a “superforecaster” would go down a bit, but the value of being “a superforecaster in expectation” would go up.
Yes—I suspect a large amount of the variance is explained by features we can measure, and the residual may be currently unexplained, but filtering on the features you can measure probably gets most of what is needed.
However, I don’t think the conclusion necessarily follows.
The problem is a causal reasoning / incentive issue (because of reasons) - just because people who update frequently do well doesn’t mean that telling people you’ll pay those who update frequently will cause them to do better now that they update more often. For instance, if you took MMORPG players and gave them money on condition that they spend money on the game, you’ll screw up the relationship between spending and success.
Fair point. I’m sure you expect some correlation between the use of reasonable incentive structures and useful updating though. It may not be perfect, but I’d be surprised if it were 0.
Agreed.