Find out the general ideological biases of the test subject
Find two studies, one (Study A) that supports the ideological biases of the test subject, but is methodologically flawed. The other (Study B) refutes the ideological biases of the subject, but is methodologically sound.
Have the subject read/research information about the studies, and then ask them which study is more correct.
If you randomize this a bit (sometimes the study is both correct and “inline with one’s bias”) and run this multiple times on a person, you should get a pretty good read on how rational they are.
Some people might decide “Because I want to show off how rational I am, I’ll accept that study X is more methodologically sound, but I’ll still believe in my secret heart that Y is correct”
I’m not sure any amount of testing can handle that much self-deception, although I’m willing to be convinced otherwise :)
How do you know your determination of “ideological bias” isn’t biased itself?
All experiments are flawed in one way or another to some degree. Are you saying one study is more methodologically flawed than another? How do you measure the degree of the flaws? How do you know your determination of flaws isn’t biased?
Again, you’ve already decided the which study is “correct” based on your own biased interpretations. How do you prove the other person is wrong and it’s not you that is biased?
I agree with the randomize and repeat bit though.
However, I would like to propose that this test methodology for rationality is deeply flawed.
Another test.
Find out the general ideological biases of the test subject
Find two studies, one (Study A) that supports the ideological biases of the test subject, but is methodologically flawed. The other (Study B) refutes the ideological biases of the subject, but is methodologically sound.
Have the subject read/research information about the studies, and then ask them which study is more correct.
If you randomize this a bit (sometimes the study is both correct and “inline with one’s bias”) and run this multiple times on a person, you should get a pretty good read on how rational they are.
Some people might decide “Because I want to show off how rational I am, I’ll accept that study X is more methodologically sound, but I’ll still believe in my secret heart that Y is correct”
I’m not sure any amount of testing can handle that much self-deception, although I’m willing to be convinced otherwise :)
How do you know your determination of “ideological bias” isn’t biased itself?
All experiments are flawed in one way or another to some degree. Are you saying one study is more methodologically flawed than another? How do you measure the degree of the flaws? How do you know your determination of flaws isn’t biased?
Again, you’ve already decided the which study is “correct” based on your own biased interpretations. How do you prove the other person is wrong and it’s not you that is biased?
I agree with the randomize and repeat bit though.
However, I would like to propose that this test methodology for rationality is deeply flawed.