That sample question reminds me of a “lie score”, which is a hidden part of some personality tests. Among the serious questions, there are also some questions like this, where you are almost certain that the “nice” answer is a lie. Most people will lie on one or two of ten such question, but the rule of thumb is that if they lie in five or more, you just throw the questionnaire away and declare them a cheater. -- However, if they didn’t lie on any of these question, you do a background check whether they have studied psychology. And you keep in mind that the test score may be manipulated.
Okay, I admit that this problem would be much worse for rationality tests, because if you want a person with given personality, they most likely didn’t study psychology. But if CFAR or similar organizations become very popular, then many candidates for highly rational people will be “tainted” by the explicit study of rationality, simply because studying rationality explicitly is probably a rational thing to do (this is just an assumption), but it’s also what an irrational person self-identifying as a rationalist would do. Also, practicing for IQ tests is obvious cheating, but practicing for getting better at doing rational tasks is the rational thing to do, and a wannabe rationalist would do it, too.
Well, seems like the rationality tests would be more similar to IQ tests than to personality test. Puzzles, time limits… maybe even the reaction times or lie detectors.
Among the serious questions, there are also some questions like this, where you are almost certain that the “nice” answer is a lie.
On the Crowne-Marlowe scale, it looks to me (having found a copy online and taken it) like most of the questions are of this form. When I answered all of the questions honestly, I scored 6, which according to the test, indicates that I am “more willing than most people to respond to tests truthfully”; but what it indicates to me is that, for all but 6 out of 33 questions, the “nice” answer was a lie, at least for me.
The 6 questions were the ones where the answer I gave was, according to the test, the “nice” one, but just happened to be the truth in my case: for example, one of the 6 was “T F I like to gossip at times”; I answered “F”, which is the “nice” answer according to the test—presumably on the assumption that most people do like to gossip but don’t want to admit it—but I genuinely don’t like to gossip at all, and can’t stand talking to people who do. Of course, now you have the problem of deciding whether that statement is true or not. :-)
Could a rationality test be gamed by lying? I think that possibility is inevitable for a test where all you can do is ask the subject questions; you always have the issue of how to know they are answering honestly.
Well, seems like the rationality tests would be more similar to IQ tests than to personality test. Puzzles, time limits… maybe even the reaction times or lie detectors.
Yes, reaction times seem like an interesting possibility. There is an online test for racism which uses this principle. But it would be pretty easy to beat the test if the results counted for anything. Actually lie detectors can be beaten too.
Perhaps brain imaging will eventually advance to the point where you can cheaply and accurately determine if someone is engaged in deception or self-deception :)
That sample question reminds me of a “lie score”, which is a hidden part of some personality tests. Among the serious questions, there are also some questions like this, where you are almost certain that the “nice” answer is a lie. Most people will lie on one or two of ten such question, but the rule of thumb is that if they lie in five or more, you just throw the questionnaire away and declare them a cheater. -- However, if they didn’t lie on any of these question, you do a background check whether they have studied psychology. And you keep in mind that the test score may be manipulated.
Okay, I admit that this problem would be much worse for rationality tests, because if you want a person with given personality, they most likely didn’t study psychology. But if CFAR or similar organizations become very popular, then many candidates for highly rational people will be “tainted” by the explicit study of rationality, simply because studying rationality explicitly is probably a rational thing to do (this is just an assumption), but it’s also what an irrational person self-identifying as a rationalist would do. Also, practicing for IQ tests is obvious cheating, but practicing for getting better at doing rational tasks is the rational thing to do, and a wannabe rationalist would do it, too.
Well, seems like the rationality tests would be more similar to IQ tests than to personality test. Puzzles, time limits… maybe even the reaction times or lie detectors.
On the Crowne-Marlowe scale, it looks to me (having found a copy online and taken it) like most of the questions are of this form. When I answered all of the questions honestly, I scored 6, which according to the test, indicates that I am “more willing than most people to respond to tests truthfully”; but what it indicates to me is that, for all but 6 out of 33 questions, the “nice” answer was a lie, at least for me.
The 6 questions were the ones where the answer I gave was, according to the test, the “nice” one, but just happened to be the truth in my case: for example, one of the 6 was “T F I like to gossip at times”; I answered “F”, which is the “nice” answer according to the test—presumably on the assumption that most people do like to gossip but don’t want to admit it—but I genuinely don’t like to gossip at all, and can’t stand talking to people who do. Of course, now you have the problem of deciding whether that statement is true or not. :-)
Could a rationality test be gamed by lying? I think that possibility is inevitable for a test where all you can do is ask the subject questions; you always have the issue of how to know they are answering honestly.
Yes, reaction times seem like an interesting possibility. There is an online test for racism which uses this principle. But it would be pretty easy to beat the test if the results counted for anything. Actually lie detectors can be beaten too.
Perhaps brain imaging will eventually advance to the point where you can cheaply and accurately determine if someone is engaged in deception or self-deception :)