The accuracy of the predictions could easily end up far more correlated with something other than predictor’s likely predictive accuracy on yet undetermined questions.
Example: I go back to mid 2000 and ask 50 Americans to make predictions about their economy, budget deficit, world standing etc of America 8 years later. The highest scorers will be pessimists, not necessarily rationalists making the best use of data available. The winner: someone who thought Al Gore was about to wreck the country.
That basically a problem that has to do with having questions where answers don’t correlate with each other.
Lets say I ask whether you think that a company will win any of the XPrizes in 2010.
I don’t necessarily think that the result of the answer correlates with optimisim and pessimism about the economy in that timeframe.
You can probably even statistically control for optimism/pessimism.
I think you’re right, if you managed to ask questions where accuracy at answering set 1 correlated very strongly with accuracy with set 2...
but didn’t correlate strongly with other factors, such as pessimism or politics...
and you manage to do that despite lots of uncertainty about those answering the questions (you’re still trying to find out about their beliefs, after all)...
If you ask people whether they believe in atheism as Eliezer suggested that also has the problem of being correlated with political beliefs.
It nearly entirely a question about what priors you have because there no other information on which you can reason.
We could make a contest of finding questions which results that don’t correlate with the other questions. Thanks to google docs quizzing people online is easy these days.
The accuracy of the predictions could easily end up far more correlated with something other than predictor’s likely predictive accuracy on yet undetermined questions.
Example: I go back to mid 2000 and ask 50 Americans to make predictions about their economy, budget deficit, world standing etc of America 8 years later. The highest scorers will be pessimists, not necessarily rationalists making the best use of data available. The winner: someone who thought Al Gore was about to wreck the country.
That basically a problem that has to do with having questions where answers don’t correlate with each other. Lets say I ask whether you think that a company will win any of the XPrizes in 2010. I don’t necessarily think that the result of the answer correlates with optimisim and pessimism about the economy in that timeframe.
You can probably even statistically control for optimism/pessimism.
I think you’re right, if you managed to ask questions where accuracy at answering set 1 correlated very strongly with accuracy with set 2...
but didn’t correlate strongly with other factors, such as pessimism or politics...
and you manage to do that despite lots of uncertainty about those answering the questions (you’re still trying to find out about their beliefs, after all)...
then you win.
If you ask people whether they believe in atheism as Eliezer suggested that also has the problem of being correlated with political beliefs. It nearly entirely a question about what priors you have because there no other information on which you can reason.
We could make a contest of finding questions which results that don’t correlate with the other questions. Thanks to google docs quizzing people online is easy these days.