Okay, I think there is a decent probability that you are right, but at this moment we need more data, which we will get by trying to create different kinds of rationality tests.
A possible outcome is that we won’t get true rationality tests, but at least something partially useful, e.g. tests selecting the people capable of rational though, which includes a lot of irrational people, but still not everyone. Which may still appear to be just another form of intelligence tests (a sufficiently intelligent irrational person is able to make rational bets, and still believe they have an invisible dragon in the garage).
So… perhaps this is a moment where I should make a bet about my beliefs. Assuming that Stanovich does not give up, and other people will follow him (that is, assuming that enough psychologists will even try to create rationality tests), I’d guess… probability 20% within 5 years, 40% within 10 years, 80% ever (pre-Singularity) that there will be a test which predicts rationality significantly better than an IQ test. Not completely reliably, but sufficiently that you would want your employees to be tested by that test instead of an IQ test, even if you had to pay more for it. (Which doesn’t mean that employers actually will want to use it. Or will be legally allowed to.) And probability 10% within 10 years, 60% ever that a true “rationality test” will be invented, at least for values up to 130 (which still many compartmentalizing people will pass). These numbers are just a wild guess, tomorrow I would probably give different values; I just thought it would be proper to express my beliefs in this format, because it encourages rationality in general.
Which may still appear to be just another form of intelligence tests (
Yes, I have a feeling that “capability of rationality” would be highly correlated with IQ.
Not completely reliably, but sufficiently that you would want your employees to be tested by that test instead of an IQ test
Your mention of employees raises another issue, which is who the test would be aimed at. When we first started discussing the issue, I had an (admittedly vague) idea in my head that the test could be for aspiring rationalists. i.e. that it could it be used to bust irrational lesswrong posters who are far less rational than they realize. It’s arguably more of a challenge to come up with a test to smoke out the self-proclaimed paragon of rationality who has the advantage of careful study and who knows exactly what he is being tested for.
By analogy, consider the Crown-Marlow Social Desirability Scale, which has been described as a test which measures “the respondent’s desire to exaggerate his own moral excellence and to present a socially desirable facade” Here is a sample question from the test:
T F I have never intensely disliked anyone
Probably the test works pretty well for your typical Joe or Jane Sixpack. But someone who is intelligent; who has studied up in this area; and who knows what’s being tested will surely conceal his desire to exaggerate his moral excellence.
That said, having thought about it, I do think there is a decent chance that solid rationality tests will be developed. At least for subjects who are unprepared. One possibility is to measure reaction times as with “Project Implicit.” Perhaps self-deception is more congnitively demanding than self-honesty and therefore a clever test might measure it. But you still might run into the problem of subconscious cheating.
Perhaps self-deception is more congnitively demanding than self-honesty and therefore a clever test might measure it.
If anything, I might expect the opposite to be true in this context. Neurotypical people have fast and frugal conformity heuristics to fall back on, while self-honestly on a lot of questions would probably take some reflection; at least, that’s true for questions that require aggregating information or assessing personality characteristics rather than coming up with a single example of something.
It’d definitely be interesting to hook someone up to a polygraph or EEG and have them take the Crowne-Marlowe Scale, though.
If anything, I might expect the opposite to be true in this context.
Well consider the hypothetical I proposed:
suppose you are having a Socratic dialogue with someone who holds irrational belief X. Instead of simply laying out your argument, you ask the person whether he agrees with Proposition Y, where Proposition Y seems pretty obvious and indisputable. Our rational person might quickly and easily agree or disagree with Y. Whereas our irrational person needs to think more carefully about Y; decide whether it might undermine his position; and if it does, construct a rationalization for rejecting Y. This difference in thinking might be measured in terms of reaction times.
See what I mean?
I do agree that in other contexts, self-deception might require less thought. e.g. spouting off the socially preferable answer to a question without really thinking about what the correct answer is.
It’d definitely be interesting to hook someone up to a polygraph or EEG and have them take the Crowne-Marlowe Scale, though.
That sample question reminds me of a “lie score”, which is a hidden part of some personality tests. Among the serious questions, there are also some questions like this, where you are almost certain that the “nice” answer is a lie. Most people will lie on one or two of ten such question, but the rule of thumb is that if they lie in five or more, you just throw the questionnaire away and declare them a cheater. -- However, if they didn’t lie on any of these question, you do a background check whether they have studied psychology. And you keep in mind that the test score may be manipulated.
Okay, I admit that this problem would be much worse for rationality tests, because if you want a person with given personality, they most likely didn’t study psychology. But if CFAR or similar organizations become very popular, then many candidates for highly rational people will be “tainted” by the explicit study of rationality, simply because studying rationality explicitly is probably a rational thing to do (this is just an assumption), but it’s also what an irrational person self-identifying as a rationalist would do. Also, practicing for IQ tests is obvious cheating, but practicing for getting better at doing rational tasks is the rational thing to do, and a wannabe rationalist would do it, too.
Well, seems like the rationality tests would be more similar to IQ tests than to personality test. Puzzles, time limits… maybe even the reaction times or lie detectors.
Among the serious questions, there are also some questions like this, where you are almost certain that the “nice” answer is a lie.
On the Crowne-Marlowe scale, it looks to me (having found a copy online and taken it) like most of the questions are of this form. When I answered all of the questions honestly, I scored 6, which according to the test, indicates that I am “more willing than most people to respond to tests truthfully”; but what it indicates to me is that, for all but 6 out of 33 questions, the “nice” answer was a lie, at least for me.
The 6 questions were the ones where the answer I gave was, according to the test, the “nice” one, but just happened to be the truth in my case: for example, one of the 6 was “T F I like to gossip at times”; I answered “F”, which is the “nice” answer according to the test—presumably on the assumption that most people do like to gossip but don’t want to admit it—but I genuinely don’t like to gossip at all, and can’t stand talking to people who do. Of course, now you have the problem of deciding whether that statement is true or not. :-)
Could a rationality test be gamed by lying? I think that possibility is inevitable for a test where all you can do is ask the subject questions; you always have the issue of how to know they are answering honestly.
Well, seems like the rationality tests would be more similar to IQ tests than to personality test. Puzzles, time limits… maybe even the reaction times or lie detectors.
Yes, reaction times seem like an interesting possibility. There is an online test for racism which uses this principle. But it would be pretty easy to beat the test if the results counted for anything. Actually lie detectors can be beaten too.
Perhaps brain imaging will eventually advance to the point where you can cheaply and accurately determine if someone is engaged in deception or self-deception :)
Okay, I think there is a decent probability that you are right, but at this moment we need more data, which we will get by trying to create different kinds of rationality tests.
A possible outcome is that we won’t get true rationality tests, but at least something partially useful, e.g. tests selecting the people capable of rational though, which includes a lot of irrational people, but still not everyone. Which may still appear to be just another form of intelligence tests (a sufficiently intelligent irrational person is able to make rational bets, and still believe they have an invisible dragon in the garage).
So… perhaps this is a moment where I should make a bet about my beliefs. Assuming that Stanovich does not give up, and other people will follow him (that is, assuming that enough psychologists will even try to create rationality tests), I’d guess… probability 20% within 5 years, 40% within 10 years, 80% ever (pre-Singularity) that there will be a test which predicts rationality significantly better than an IQ test. Not completely reliably, but sufficiently that you would want your employees to be tested by that test instead of an IQ test, even if you had to pay more for it. (Which doesn’t mean that employers actually will want to use it. Or will be legally allowed to.) And probability 10% within 10 years, 60% ever that a true “rationality test” will be invented, at least for values up to 130 (which still many compartmentalizing people will pass). These numbers are just a wild guess, tomorrow I would probably give different values; I just thought it would be proper to express my beliefs in this format, because it encourages rationality in general.
Yes, I have a feeling that “capability of rationality” would be highly correlated with IQ.
Your mention of employees raises another issue, which is who the test would be aimed at. When we first started discussing the issue, I had an (admittedly vague) idea in my head that the test could be for aspiring rationalists. i.e. that it could it be used to bust irrational lesswrong posters who are far less rational than they realize. It’s arguably more of a challenge to come up with a test to smoke out the self-proclaimed paragon of rationality who has the advantage of careful study and who knows exactly what he is being tested for.
By analogy, consider the Crown-Marlow Social Desirability Scale, which has been described as a test which measures “the respondent’s desire to exaggerate his own moral excellence and to present a socially desirable facade” Here is a sample question from the test:
Probably the test works pretty well for your typical Joe or Jane Sixpack. But someone who is intelligent; who has studied up in this area; and who knows what’s being tested will surely conceal his desire to exaggerate his moral excellence.
That said, having thought about it, I do think there is a decent chance that solid rationality tests will be developed. At least for subjects who are unprepared. One possibility is to measure reaction times as with “Project Implicit.” Perhaps self-deception is more congnitively demanding than self-honesty and therefore a clever test might measure it. But you still might run into the problem of subconscious cheating.
If anything, I might expect the opposite to be true in this context. Neurotypical people have fast and frugal conformity heuristics to fall back on, while self-honestly on a lot of questions would probably take some reflection; at least, that’s true for questions that require aggregating information or assessing personality characteristics rather than coming up with a single example of something.
It’d definitely be interesting to hook someone up to a polygraph or EEG and have them take the Crowne-Marlowe Scale, though.
Well consider the hypothetical I proposed:
See what I mean?
I do agree that in other contexts, self-deception might require less thought. e.g. spouting off the socially preferable answer to a question without really thinking about what the correct answer is.
Yes.
That sample question reminds me of a “lie score”, which is a hidden part of some personality tests. Among the serious questions, there are also some questions like this, where you are almost certain that the “nice” answer is a lie. Most people will lie on one or two of ten such question, but the rule of thumb is that if they lie in five or more, you just throw the questionnaire away and declare them a cheater. -- However, if they didn’t lie on any of these question, you do a background check whether they have studied psychology. And you keep in mind that the test score may be manipulated.
Okay, I admit that this problem would be much worse for rationality tests, because if you want a person with given personality, they most likely didn’t study psychology. But if CFAR or similar organizations become very popular, then many candidates for highly rational people will be “tainted” by the explicit study of rationality, simply because studying rationality explicitly is probably a rational thing to do (this is just an assumption), but it’s also what an irrational person self-identifying as a rationalist would do. Also, practicing for IQ tests is obvious cheating, but practicing for getting better at doing rational tasks is the rational thing to do, and a wannabe rationalist would do it, too.
Well, seems like the rationality tests would be more similar to IQ tests than to personality test. Puzzles, time limits… maybe even the reaction times or lie detectors.
On the Crowne-Marlowe scale, it looks to me (having found a copy online and taken it) like most of the questions are of this form. When I answered all of the questions honestly, I scored 6, which according to the test, indicates that I am “more willing than most people to respond to tests truthfully”; but what it indicates to me is that, for all but 6 out of 33 questions, the “nice” answer was a lie, at least for me.
The 6 questions were the ones where the answer I gave was, according to the test, the “nice” one, but just happened to be the truth in my case: for example, one of the 6 was “T F I like to gossip at times”; I answered “F”, which is the “nice” answer according to the test—presumably on the assumption that most people do like to gossip but don’t want to admit it—but I genuinely don’t like to gossip at all, and can’t stand talking to people who do. Of course, now you have the problem of deciding whether that statement is true or not. :-)
Could a rationality test be gamed by lying? I think that possibility is inevitable for a test where all you can do is ask the subject questions; you always have the issue of how to know they are answering honestly.
Yes, reaction times seem like an interesting possibility. There is an online test for racism which uses this principle. But it would be pretty easy to beat the test if the results counted for anything. Actually lie detectors can be beaten too.
Perhaps brain imaging will eventually advance to the point where you can cheaply and accurately determine if someone is engaged in deception or self-deception :)