However, it seems to me that the fatal problem of the sampling theory is that nobody has ever managed to figure out a way to sample disjoint sets of these hypothetical uncorrelated modules. If all practically useful mental abilities and all the tests successfully predicting them always sample some particular subset of these modules, then we might as well look at that subset as a unified entity that represents the causal factor behind g, since its elements operate together as a group in all relevant cases.
Or is there some additional issue here that I’m not taking into account?
I can’t immediately think of any additional issue. It’s more that I don’t see the lack of well-known disjoint sets of uncorrelated cognitive modules as a fatal problem for Thomson’s theory, merely weak disconfirming evidence. This is because I assign a relatively low probability to psychologists detecting tests that sample disjoint sets of modules even if they exist.
For example, I can think of a situation where psychologists & psychometricians have missed a similar phenomenon: negatively correlated cognitive tests. I know of a couple of examples which I found only because the mathematician Warren D. Smith describes them in his paper “Mathematical definition of ‘intelligence’ (and consequences)”. The paper’s about the general goal of coming up with universal definitions of and ways to measure intelligence, but in the middle of it is a polemical/sceptical summary of research into g & IQ.
Smith went through a correlation matrix for 57 tests given to 240 people, published by Thurstone in 1938, and saw that the 3 most negative of the 1596 intercorrelations were between these pairs of tests:
“100-word vocabulary test // Recognize pictures of hand as Right/Left” (correlation = −0.22)
“Find lots of synonyms of a given word // Decide whether 2 pictures of a national flag are relatively mirrored or not” (correlation = −0.16)
“Describe somebody in writing: score=# words used // figure recognition test: decide which numbers in a list of drawings of abstract figures are ones you saw in a previously shown list” (correlation = −0.12)
In Smith’s words: “This seems too much to be a coincidence!” Smith then went to the 60-item correlation matrix for 710 schoolchildren published by Thurstone & Thurstone in 1941 and did the same, discovering that
the three most negative [correlations], with values −0.161, −0.152, and −0.138 respectively, are the pairwise correlations of the performance on the “scattered Xs” test (circle the Xs in a random scattering of letters) with these three tests: (a) Sentence completion … (b) Reading comprehension II … (c) Reading comprehension I … Again, it is difficult to believe this also is a coincidence!
The existence of two pairs of negatively correlated cognitive skills leads me to increase my prior for the existence of uncorrelated cognitive skills.
Also, the way psychologists often analyze test batteries makes it harder to spot disjoint sets of uncorrelated modules. Suppose we have a 3-test battery, where test 1 samples uncorrelated modules A, B, C, D & E, test 2 samples F, G, H, I & J, and test 3 samples C, D, E, F & G. If we administer the battery to a few thousand people and extract a g from the results, as is standard practice, then by construction the resulting g is going to correlate with scores on tests 1 & 2, although we know they sample non-overlapping sets of modules. (IQ, being a weighted average of test/module scores, will also correlate with all of the tests.) A lot of psychologists would interpret that as evidence against tests 1 & 2 measuring distinct mental abilities, even though we see there’s an alternative explanation.
Even if we did find an index of intelligence that didn’t correlate with IQ/g, would we count it as such? Duckworth & Seligman discovered that in a sample of 164 schoolchildren, a composite measure of self-discipline predicted GPA significantly better than IQ, and self-discipline didn’t correlate significantly with IQ. Does self-discipline now count as an independent intellectual ability? I’d lean towards saying it doesn’t, but I doubt I could justify being dogmatic about that; it’s surely a cognitive ability in the term’s broadest sense.
For example, I can think of a situation where psychologists & psychometricians have missed a similar phenomenon: negatively correlated cognitive tests. I know of a couple of examples which I found only because the mathematician Warren D. Smith describes them in his paper “Mathematical definition of ‘intelligence’ (and consequences)”.
That’s an extremely interesting reference, thanks for the link! This is exactly the kind of approach that this area desperately needs: no-nonsense scrutiny by someone with a strong math background and without an ideological agenda.
David Hilbert allegedly once quipped that physics is too important to be left to physicists; the way things are, it seems to me that psychometrics should definitely not be left to psychologists. That they haven’t immediately rushed to explore further these findings by Smith is an extremely damning fact about the intellectual standards in the field.
Duckworth & Seligman discovered that in a sample of 164 schoolchildren, a composite measure of self-discipline predicted GPA significantly better than IQ, and self-discipline didn’t correlate significantly with IQ. Does self-discipline now count as an independent intellectual ability?
Wouldn’t this closely correspond to the Big Five “conscientiousness” trait? (Which the paper apparently doesn’t mention at all?!) From what I’ve seen, even among the biggest fans of IQ, it is generally recognized that conscientiousness is at least similarly important as general intelligence in predicting success and performance.
Wouldn’t this closely correspond to the Big Five “conscientiousness” trait? (Which the paper apparently doesn’t mention at all?!) From what I’ve seen, even among the biggest fans of IQ, it is generally recognized that conscientiousness is at least similarly important as general intelligence in predicting success and performance.
That’s an excellent point that completely did not occur to me. Turns out that self-discipline is actually one of the 6 subscales used to measure conscientiousness on the NEO-PI-R, so it’s clearly related to conscientiousness. With that in mind, it is a bit weird that conscientiousness doesn’t get a shoutout in the paper...
Is anything known about a physical basis for conscientiousness?
It can be reliably predicted by, for example, SPECT scans. If I recall correctly you can expect to see over-active frontal lobes and basal ganglia. For this reason (and because those areas depend on dopamine a lot) dopaminergics (Ritalin, etc) make a big difference.
I haven’t looked at Smith yet, but the quote looks like parody to me. Since you seem to take it seriously, I’ll respond. Awfully specific tests defying the predictions looks like data mining to me. I predict that these negative correlations are not replicable. The first seems to be the claim that verbal ability is not correlated with spatial ability, but this is a well-tested claim. As Shalizi mentions, psychometricians do look for separate skills and these are commonly accepted components. I wouldn’t be terribly surprised if there were ones they completely missed, but these two are popular and positively correlated. The second example is a little more promising: maybe that scattered Xs test is independent of verbal ability, even though it looks like other skills that are not, but I doubt it.
With respect to self-discipline, I think you’re experiencing some kind of halo effect. Not every positive mental trait should be called intelligence. Self-discipline is just not what people mean by intelligence. I knew that conscientiousness predicted GPAs, but I’d never heard such a strong claim. But it is true that a lot of people dismiss conscientiousness (and GPA) in favor of intelligence, and they seem to be making an error (or being risk-seeking).
I haven’t looked at Smith yet, but the quote looks like parody to me. Since you seem to take it seriously, I’ll respond.
Once you read the relevant passage in context, I anticipate you will agree with me that Smith is serious. Take this paragraph from before the passage I quoted from:
Further, let us return to Gould’s criticism that due to “validation” of most other highly used IQ tests and subtests, Spearman’s g was forced to appear to exist from then on, regardless of whether it actually did. In view of this … probably the only place we can look in the literature to find data truly capable of refuting or confirming Spearman, is data from the early days, before too much “validation” occurred, but not so early on that Spearman’s atrocious experimental and statistical practices were repeated.
The prime candidate I have been able to find for such data is Thurstone’s [205] “primary mental abilities” dataset published in 1938.
Smith then presents the example from Thurstone’s 1938 data.
Awfully specific tests defying the predictions looks like data mining to me. I predict that these negative correlations are not replicable.
I’d be inclined to agree if the 3 most negative correlations in the dataset had come from very different pairs of tests, but the fact that they come from sets of subtests that one would expect to tap similar narrow abilities suggests they’re not just statistical noise.
The first seems to be the claim that verbal ability is not correlated with spatial ability, but this is a well-tested claim. As Shalizi mentions, psychometricians do look for separate skills and these are commonly accepted components. I wouldn’t be terribly surprised if there were ones they completely missed, but these two are popular and positively correlated.
Smith himself does not appear to make that claim; he presents his two examples merely as demonstrations that not all mental ability scores positively correlate. I think it’s reasonable to package the 3 verbal subtests he mentions as strongly loading on verbal ability, but it’s not clear to me that the 3 other subtests he pairs them with are strong measures of “spatial ability”; two of them look like they tap a more specific ability to handle mental mirror images, and the third’s a visual memory test.
Even if it transpires that the 3 subtests all tap substantially into spatial ability, they needn’t necessarily correlate positively with specific measures of verbal ability, even though verbal ability correlates with spatial ability.
With respect to self-discipline, I think you’re experiencing some kind of halo effect. Not every positive mental trait should be called intelligence. Self-discipline is just not what people mean by intelligence.
I’m tempted to agree but I’m not sure such a strong generalization is defensible. Take a list of psychologists’ definitions of intelligence. IMO self-discipline plausibly makes sense as a component of intelligence under definitions 1, 7, 8, 13, 14, 23, 25, 26, 27, 28, 32, 33 & 34, which adds up to 37% of the list of definitions. A good few psychologists appear to include self-discipline as a facet of intelligence.
Even if we did find an index of intelligence that didn’t correlate with IQ/g, would we count it as such? Duckworth & Seligman discovered that in a sample of 164 schoolchildren, a composite measure of self-discipline predicted GPA significantly better than IQ, and self-discipline didn’t correlate significantly with IQ. Does self-discipline now count as an independent intellectual ability? I’d lean towards saying it doesn’t, but I doubt I could justify being dogmatic about that; it’s surely a cognitive ability in the term’s broadest sense.
This finding is consistent with the folk notion of “crazy geniuses.”
Though it’s important to note that the second study was done on college students, who must have a certain level of IQ and who aren’t representative of the population.
The first study notes:
According to this proposal the significant negative correlation could be observed only in groups with above average mental abilities and not in a random sample from a general population.
If we took a larger sample of the population, including lower IQ individuals, then I think we would see the negative correlation between Conscientiousness and intelligence diminish or even reverse, because I bet there are lots of people outside a college population who have both low intelligence and low Conscientiousness.
It could be that a moderate amount of Conscientiousness (well, whatever mechanisms cause Conscientiousness) is necessary for above average intelligence, but too much Conscientiousness (i.e. those mechanisms are too strong) limits intelligence.
I noticed a while back when a bunch of LW’ers gave their Big Five scores that our Conscientiousness scores tended to be low. I took that to be an internet thing (people currently reading a website are more likely to be lazy slobs) but this is a more flattering explanation.
Interesting. I would’ve expected Conscientiousness to correlate weakly positively with IQ across most IQ levels.
I would avoid interpreting a negative correlation between C/self-discipline and IQ as evidence against C/self-discipline being a separate facet of intelligence; I think that would beg the question by implicitly assuming that IQ’s representing the entirety of what we call intelligence.
However, it seems to me that the fatal problem of the sampling theory is that nobody has ever managed to figure out a way to sample disjoint sets of these hypothetical uncorrelated modules. If all practically useful mental abilities and all the tests successfully predicting them always sample some particular subset of these modules, then we might as well look at that subset as a unified entity that represents the causal factor behind g, since its elements operate together as a group in all relevant cases.
Or is there some additional issue here that I’m not taking into account?
I can’t immediately think of any additional issue. It’s more that I don’t see the lack of well-known disjoint sets of uncorrelated cognitive modules as a fatal problem for Thomson’s theory, merely weak disconfirming evidence. This is because I assign a relatively low probability to psychologists detecting tests that sample disjoint sets of modules even if they exist.
For example, I can think of a situation where psychologists & psychometricians have missed a similar phenomenon: negatively correlated cognitive tests. I know of a couple of examples which I found only because the mathematician Warren D. Smith describes them in his paper “Mathematical definition of ‘intelligence’ (and consequences)”. The paper’s about the general goal of coming up with universal definitions of and ways to measure intelligence, but in the middle of it is a polemical/sceptical summary of research into g & IQ.
Smith went through a correlation matrix for 57 tests given to 240 people, published by Thurstone in 1938, and saw that the 3 most negative of the 1596 intercorrelations were between these pairs of tests:
“100-word vocabulary test // Recognize pictures of hand as Right/Left” (correlation = −0.22)
“Find lots of synonyms of a given word // Decide whether 2 pictures of a national flag are relatively mirrored or not” (correlation = −0.16)
“Describe somebody in writing: score=# words used // figure recognition test: decide which numbers in a list of drawings of abstract figures are ones you saw in a previously shown list” (correlation = −0.12)
In Smith’s words: “This seems too much to be a coincidence!” Smith then went to the 60-item correlation matrix for 710 schoolchildren published by Thurstone & Thurstone in 1941 and did the same, discovering that
The existence of two pairs of negatively correlated cognitive skills leads me to increase my prior for the existence of uncorrelated cognitive skills.
Also, the way psychologists often analyze test batteries makes it harder to spot disjoint sets of uncorrelated modules. Suppose we have a 3-test battery, where test 1 samples uncorrelated modules A, B, C, D & E, test 2 samples F, G, H, I & J, and test 3 samples C, D, E, F & G. If we administer the battery to a few thousand people and extract a g from the results, as is standard practice, then by construction the resulting g is going to correlate with scores on tests 1 & 2, although we know they sample non-overlapping sets of modules. (IQ, being a weighted average of test/module scores, will also correlate with all of the tests.) A lot of psychologists would interpret that as evidence against tests 1 & 2 measuring distinct mental abilities, even though we see there’s an alternative explanation.
Even if we did find an index of intelligence that didn’t correlate with IQ/g, would we count it as such? Duckworth & Seligman discovered that in a sample of 164 schoolchildren, a composite measure of self-discipline predicted GPA significantly better than IQ, and self-discipline didn’t correlate significantly with IQ. Does self-discipline now count as an independent intellectual ability? I’d lean towards saying it doesn’t, but I doubt I could justify being dogmatic about that; it’s surely a cognitive ability in the term’s broadest sense.
satt:
That’s an extremely interesting reference, thanks for the link! This is exactly the kind of approach that this area desperately needs: no-nonsense scrutiny by someone with a strong math background and without an ideological agenda.
David Hilbert allegedly once quipped that physics is too important to be left to physicists; the way things are, it seems to me that psychometrics should definitely not be left to psychologists. That they haven’t immediately rushed to explore further these findings by Smith is an extremely damning fact about the intellectual standards in the field.
Wouldn’t this closely correspond to the Big Five “conscientiousness” trait? (Which the paper apparently doesn’t mention at all?!) From what I’ve seen, even among the biggest fans of IQ, it is generally recognized that conscientiousness is at least similarly important as general intelligence in predicting success and performance.
That’s an excellent point that completely did not occur to me. Turns out that self-discipline is actually one of the 6 subscales used to measure conscientiousness on the NEO-PI-R, so it’s clearly related to conscientiousness. With that in mind, it is a bit weird that conscientiousness doesn’t get a shoutout in the paper...
Is anything known about a physical basis for conscientiousness?
It can be reliably predicted by, for example, SPECT scans. If I recall correctly you can expect to see over-active frontal lobes and basal ganglia. For this reason (and because those areas depend on dopamine a lot) dopaminergics (Ritalin, etc) make a big difference.
I haven’t looked at Smith yet, but the quote looks like parody to me. Since you seem to take it seriously, I’ll respond. Awfully specific tests defying the predictions looks like data mining to me. I predict that these negative correlations are not replicable. The first seems to be the claim that verbal ability is not correlated with spatial ability, but this is a well-tested claim. As Shalizi mentions, psychometricians do look for separate skills and these are commonly accepted components. I wouldn’t be terribly surprised if there were ones they completely missed, but these two are popular and positively correlated. The second example is a little more promising: maybe that scattered Xs test is independent of verbal ability, even though it looks like other skills that are not, but I doubt it.
With respect to self-discipline, I think you’re experiencing some kind of halo effect. Not every positive mental trait should be called intelligence. Self-discipline is just not what people mean by intelligence. I knew that conscientiousness predicted GPAs, but I’d never heard such a strong claim. But it is true that a lot of people dismiss conscientiousness (and GPA) in favor of intelligence, and they seem to be making an error (or being risk-seeking).
Once you read the relevant passage in context, I anticipate you will agree with me that Smith is serious. Take this paragraph from before the passage I quoted from:
Smith then presents the example from Thurstone’s 1938 data.
I’d be inclined to agree if the 3 most negative correlations in the dataset had come from very different pairs of tests, but the fact that they come from sets of subtests that one would expect to tap similar narrow abilities suggests they’re not just statistical noise.
Smith himself does not appear to make that claim; he presents his two examples merely as demonstrations that not all mental ability scores positively correlate. I think it’s reasonable to package the 3 verbal subtests he mentions as strongly loading on verbal ability, but it’s not clear to me that the 3 other subtests he pairs them with are strong measures of “spatial ability”; two of them look like they tap a more specific ability to handle mental mirror images, and the third’s a visual memory test.
Even if it transpires that the 3 subtests all tap substantially into spatial ability, they needn’t necessarily correlate positively with specific measures of verbal ability, even though verbal ability correlates with spatial ability.
I’m tempted to agree but I’m not sure such a strong generalization is defensible. Take a list of psychologists’ definitions of intelligence. IMO self-discipline plausibly makes sense as a component of intelligence under definitions 1, 7, 8, 13, 14, 23, 25, 26, 27, 28, 32, 33 & 34, which adds up to 37% of the list of definitions. A good few psychologists appear to include self-discipline as a facet of intelligence.
Interesting thought. It turns out that Conscientiousness is actually negatively related to intelligence, while Openness is positively correlated with intelligence.
This finding is consistent with the folk notion of “crazy geniuses.”
Though it’s important to note that the second study was done on college students, who must have a certain level of IQ and who aren’t representative of the population.
The first study notes:
If we took a larger sample of the population, including lower IQ individuals, then I think we would see the negative correlation between Conscientiousness and intelligence diminish or even reverse, because I bet there are lots of people outside a college population who have both low intelligence and low Conscientiousness.
It could be that a moderate amount of Conscientiousness (well, whatever mechanisms cause Conscientiousness) is necessary for above average intelligence, but too much Conscientiousness (i.e. those mechanisms are too strong) limits intelligence.
I noticed a while back when a bunch of LW’ers gave their Big Five scores that our Conscientiousness scores tended to be low. I took that to be an internet thing (people currently reading a website are more likely to be lazy slobs) but this is a more flattering explanation.
No it doesn’t. The whole point of that article is that it’s a mistake to ask people how conscientious they are.
Interesting. I would’ve expected Conscientiousness to correlate weakly positively with IQ across most IQ levels.
I would avoid interpreting a negative correlation between C/self-discipline and IQ as evidence against C/self-discipline being a separate facet of intelligence; I think that would beg the question by implicitly assuming that IQ’s representing the entirety of what we call intelligence.