I think you’re right, if you managed to ask questions where accuracy at answering set 1 correlated very strongly with accuracy with set 2...
but didn’t correlate strongly with other factors, such as pessimism or politics...
and you manage to do that despite lots of uncertainty about those answering the questions (you’re still trying to find out about their beliefs, after all)...
If you ask people whether they believe in atheism as Eliezer suggested that also has the problem of being correlated with political beliefs.
It nearly entirely a question about what priors you have because there no other information on which you can reason.
We could make a contest of finding questions which results that don’t correlate with the other questions. Thanks to google docs quizzing people online is easy these days.
I think you’re right, if you managed to ask questions where accuracy at answering set 1 correlated very strongly with accuracy with set 2...
but didn’t correlate strongly with other factors, such as pessimism or politics...
and you manage to do that despite lots of uncertainty about those answering the questions (you’re still trying to find out about their beliefs, after all)...
then you win.
If you ask people whether they believe in atheism as Eliezer suggested that also has the problem of being correlated with political beliefs. It nearly entirely a question about what priors you have because there no other information on which you can reason.
We could make a contest of finding questions which results that don’t correlate with the other questions. Thanks to google docs quizzing people online is easy these days.