The self-reported IQ results on these surveys have been, to use Yvain’s wording, “ridiculed” because they’d mean that the average LessWronger is gifted. Various other questions were added to the survey this time which gives us things to check against, and the results of these other questions have made the IQ figures more believable.
Summary:
LessWrong has lost IQ points on the self-reported scores every year for a total of 7.18 IQ points in 3.7 years or about 2 points per year. If LessWrong began with 145.88 IQ points in May 2009, then LessWrong has lost over half of it’s giftedness (using IQ 132 as the definition, explained below).
* Estimated IQ point drops for 2010 and 2011: I divided the 2011 IQ drop by 2 and distributed it across 10⁄11.
* IQ 132 significance: IQ 132 is the top 2% (This may vary a little bit from one IQ test to another) which would qualify one as gifted by every IQ-based definition I know of. It is also (roughly) Mensa’s entrance requirement (depending on the test) though Mensa does not dictate the legal or psychologist’s definitions of giftedness. They are a club, not a developmental psychology authority.
As I mentioned previously, and judging from the graphs, the standard deviations of the IQs are obviously mixed up, because they were not determined in the questionnaire, and probably people who answered are not educated about them either. Including IQs in s.d. 24 with those in s.d. 16 and 15 is bound to inflate the average IQ. The top scores in that graph, or at the very least some of them, are in s.d. 24, which means that they would be a lot lower in s.d. 15. IQ 132 is the cutoff for s.d. 16, while s.d. 15 is the one most adopted in recent scientific literature. For s.d. 24, it is 148. Mensa and often people on the press like to use s.d. 24 to sound more impressive to amateurs.
This probably makes tests like the SAT more reliable as an estimation, because they have the same standard for all who submitted their scores, although in this case the ceiling effect would become apparent, because perfect or nearly-perfect scores wouldn’t go upwards of a certain IQ.
The basic formula for a confidence interval of a population is: mean ± (z-score of confidence × (standard deviation / √n)). So for z-score=95%=1.96:
= the range 142.5-149.2
= the range 141.5-138.7
= the range 137-139.6
Or to run the usual t-tests and look at the confidence interval they calculate for the difference; for 2009 & 2012, the 95% CI for the difference in mean IQ is 3.563-10.578:
R> lw2009 <- read.csv("lw-2009.csv")
R> lw2011 <- read.csv("lw-2011.csv")
R> lw2012 <- read.csv("lw-2012.csv")
R> # lwi2009 <- lw2009$IQ[!is.na(lw2009$IQ)]
R> # hand-cleaned:
R> lwi2009 <- c(120,125,128,129,130,130,130,130,130,130,130,130,130,131,132,132,133,134,136,138,138,139,139,140,
140,140,140,140,140,140,140,140,140,141,142,144,145,145,145,148,148,150,150,150,150,152,154,154,
155,155,155,155,156,158,158,160,160,160,160,162,163,164,165,166,170,171,173,180)
R> lwi2011 <- lw2011$IQ[!is.na(lw2011$IQ)]
R> lwi2012 <- lw2012$IQ[!is.na(lw2012$IQ)]
R>
R> t.test(lwi2009, lwi2012)
Welch Two Sample t-test
data: lwi2009 and lwi2012
t = 4.004, df = 91.49, p-value = 0.0001264
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
3.563 10.578
sample estimates:
mean of x mean of y
145.4 138.3
R> t.test(lwi2009, lwi2011)
Welch Two Sample t-test
data: lwi2009 and lwi2011
t = 2.968, df = 94.8, p-value = 0.003791
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
1.752 8.830
sample estimates:
mean of x mean of y
145.4 140.1
R> t.test(lwi2011, lwi2012)
Welch Two Sample t-test
data: lwi2011 and lwi2012
t = 1.804, df = 670.4, p-value = 0.07174
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-0.1578 3.7174
sample estimates:
mean of x mean of y
140.1 138.3
Note that Epiphany dates the 2009 survey to around March, while the other two surveys happened around November, so inputting the survey dates just as years lowballs the time gap between the first & second surveys. Your linear trend’ll be a bit exaggerated.
Before, the slope per year was −2.24 (minus 2.25 points a year), now the slope spits out as −0.00519 but if I’m understanding my changes right, the unit has switched from per year to per day and 365.25 times −0.005 IQ points per day is −1.896 per year.
This comment is relevant; we have a dataset of users who both took the Raven’s test and self-reported IQ. The means of the group that did both was rather close to the means of the group that did each separately, but the correlation between the tests was low at .2. If you looked just at responders with positive karma, the correlation increased to a more respectable .45; if you looked just responders without positive karma, the correlation was -.11. This was a small fraction of responders as a whole, and the average IQ is already tremendously inflated by nonresponse. (If we assumed that, on average, people who didn’t self-report an IQ were IQ 100, then the LW average would be only 112!)
IQ Trend Analysis:
The self-reported IQ results on these surveys have been, to use Yvain’s wording, “ridiculed” because they’d mean that the average LessWronger is gifted. Various other questions were added to the survey this time which gives us things to check against, and the results of these other questions have made the IQ figures more believable.
Summary:
LessWrong has lost IQ points on the self-reported scores every year for a total of 7.18 IQ points in 3.7 years or about 2 points per year. If LessWrong began with 145.88 IQ points in May 2009, then LessWrong has lost over half of it’s giftedness (using IQ 132 as the definition, explained below).
The self-reported figures for each year:
IQ on 03/12/2009: 145.88
IQ on 00/00/2010: Unknown*
IQ on 12/05/2011: 140
IQ on 11/29/2012: 138.7
IQ points lost each year:
2.94 IQ point drop for 2010 (Estimated*)
2.94 IQ point drop for 2011 (Estimated*)
1.30 IQ point drop for 2012
Analysis:
Average IQ points lost per year: 1.94
Total IQ points lost: 7.18 in 3.7 years
Total IQ points LessWrong had above the gifted line: 13.88 (145.88 − 132*)
Percent less giftedness on the last survey result: 52% (7.18 / 13.88)
Footnotes:
* Unknown 2010 figures: There was no 2010 survey. The first line of the 2011 survey proposition mentions that.
* Estimated IQ point drops for 2010 and 2011: I divided the 2011 IQ drop by 2 and distributed it across 10⁄11.
* IQ 132 significance: IQ 132 is the top 2% (This may vary a little bit from one IQ test to another) which would qualify one as gifted by every IQ-based definition I know of. It is also (roughly) Mensa’s entrance requirement (depending on the test) though Mensa does not dictate the legal or psychologist’s definitions of giftedness. They are a club, not a developmental psychology authority.
As I mentioned previously, and judging from the graphs, the standard deviations of the IQs are obviously mixed up, because they were not determined in the questionnaire, and probably people who answered are not educated about them either. Including IQs in s.d. 24 with those in s.d. 16 and 15 is bound to inflate the average IQ. The top scores in that graph, or at the very least some of them, are in s.d. 24, which means that they would be a lot lower in s.d. 15. IQ 132 is the cutoff for s.d. 16, while s.d. 15 is the one most adopted in recent scientific literature. For s.d. 24, it is 148. Mensa and often people on the press like to use s.d. 24 to sound more impressive to amateurs.
This probably makes tests like the SAT more reliable as an estimation, because they have the same standard for all who submitted their scores, although in this case the ceiling effect would become apparent, because perfect or nearly-perfect scores wouldn’t go upwards of a certain IQ.
Ooh, you bring up good points. These are a source of noise, for sure.
Now I’m wondering if there are any clever ways to compensate for any of these and remove that noise from the survey…
Error bars, please!
The summary data:
2009: n=67, 145.88(14.02)
2011: n=331; 140.10(13.07)
2012: n=346; 138.30(12.58); graphed:
The basic formula for a confidence interval of a population is:
mean ± (z-score of confidence × (standard deviation / √n)). So for z-score=95%=1.96:Or to run the usual t-tests and look at the confidence interval they calculate for the difference; for 2009 & 2012, the 95% CI for the difference in mean IQ is 3.563-10.578:
To add a linear model (for those unfamiliar, see my HPMoR examples) which will really just recapitulate the simple averages calculation:
Note that Epiphany dates the 2009 survey to around March, while the other two surveys happened around November, so inputting the survey dates just as years lowballs the time gap between the first & second surveys. Your linear trend’ll be a bit exaggerated.
I’ve fixed it as appropriate.
Before, the slope per year was −2.24 (minus 2.25 points a year), now the slope spits out as −0.00519 but if I’m understanding my changes right, the unit has switched from per year to per day and 365.25 times −0.005 IQ points per day is −1.896 per year.
2.25 vs 1.9 is fairly different.
I was lazy and ignored all non-numerical IQ comments, so I got slightly different numbers. But my 95% confidence intervals are:
145.18±3.27 in 2009
140.12±1.41 in 2011
138.42±1.33 in 2012
This comment is relevant; we have a dataset of users who both took the Raven’s test and self-reported IQ. The means of the group that did both was rather close to the means of the group that did each separately, but the correlation between the tests was low at .2. If you looked just at responders with positive karma, the correlation increased to a more respectable .45; if you looked just responders without positive karma, the correlation was -.11. This was a small fraction of responders as a whole, and the average IQ is already tremendously inflated by nonresponse. (If we assumed that, on average, people who didn’t self-report an IQ were IQ 100, then the LW average would be only 112!)