That basically a problem that has to do with having questions where answers don’t correlate with each other.
Lets say I ask whether you think that a company will win any of the XPrizes in 2010.
I don’t necessarily think that the result of the answer correlates with optimisim and pessimism about the economy in that timeframe.
You can probably even statistically control for optimism/pessimism.
I think you’re right, if you managed to ask questions where accuracy at answering set 1 correlated very strongly with accuracy with set 2...
but didn’t correlate strongly with other factors, such as pessimism or politics...
and you manage to do that despite lots of uncertainty about those answering the questions (you’re still trying to find out about their beliefs, after all)...
If you ask people whether they believe in atheism as Eliezer suggested that also has the problem of being correlated with political beliefs.
It nearly entirely a question about what priors you have because there no other information on which you can reason.
We could make a contest of finding questions which results that don’t correlate with the other questions. Thanks to google docs quizzing people online is easy these days.
That basically a problem that has to do with having questions where answers don’t correlate with each other. Lets say I ask whether you think that a company will win any of the XPrizes in 2010. I don’t necessarily think that the result of the answer correlates with optimisim and pessimism about the economy in that timeframe.
You can probably even statistically control for optimism/pessimism.
I think you’re right, if you managed to ask questions where accuracy at answering set 1 correlated very strongly with accuracy with set 2...
but didn’t correlate strongly with other factors, such as pessimism or politics...
and you manage to do that despite lots of uncertainty about those answering the questions (you’re still trying to find out about their beliefs, after all)...
then you win.
If you ask people whether they believe in atheism as Eliezer suggested that also has the problem of being correlated with political beliefs. It nearly entirely a question about what priors you have because there no other information on which you can reason.
We could make a contest of finding questions which results that don’t correlate with the other questions. Thanks to google docs quizzing people online is easy these days.