LW would have roughly 25 times the baseline AS rate.
The Less Wrong mean average AQ test score was 27
Only about 1 in 10 people on Less Wrong are “normal” in terms of the empathizing/systematizing scale.
This is grossly overstated, because the survey respondents are in no way known to be representative of LessWrong. In fact, I would suspect a very strong selection bias. Before you can go saying anything about the LessWrong community, you need some response statistics. How many people were offered the survey? (You could get this from the server logs. Your survey wasn’t promoted and was only up for a day or so, an exposure that already subsets its audience from “LessWrong”) How many of the people who saw the survey decided to fill it out? How do the people who filled out the survey differ from the people who did not fill out the survey? I doubt the answers to any of these questions would be satisfying unless you started with a random sample, then individually and privately invited each of the sampled people to participate. Even then, the response bias might be bad enough to preclude any conclusions about LessWrong.
This is a good point, and we should consider it if we design a poll feature for LW, for example the poll could be made available only to registered users, and could track which of them saw it, and perhaps even give Karma rewards to those users who complete it to ensure a higher response rate.
Just out of interest, what’s your prediction of the sample mean AQ for those responses to my survey that are submitted after I put the note at the top of the post asking only previous non-responders to respond, assuming I get more than, say, 5?
Based mostly on Unnamed’s comment regarding a poll on homosexuality, I expect it would be lower than what you found for self-selected interested respondents, but I don’t have much confidence in this, and I’m not familiar enough with AQ variance to say how much lower.
I have some experience with trying to do survey research right (i.e. in a manner that results in true conclusions about the population), and I have seen so many ways that convenience samples can make me wrong (non-response bias is just a start), that I generally don’t bother much with trying to correct for one particular bias. Instead, I stick to random samples or use estimators that can handle convenience samples. But I am genuinely intrigued by your idea (from the very end of your post) to use a prior on non-response bias and a prior on AQ distribution to estimate the true AQ distribution of LW users. This wouldn’t handle e.g. problems with question wording or lies in responses, but I would expect those problems to be small in comparison to the non-response bias. Could you elaborate, or point me to any survey methods papers that explain this approach in detail? Where would you get the priors?
Even then, the response bias might be bad enough to preclude any conclusions about LessWrong.
Yes, this is a good point.
Note that it’s still evidence, in the sense that changing the results of the survey ought to change your opinions about the AS-spectrum average over the cluster of Less Wrong readers.
Of course there is a potentially huge response bias—according to sitemeter there were 3000 visitors today, meaning that the response bias could in theory be up to 60 x . (which means that with maximum response bias, i.e. all nonresponders are scoring an average of 16 on the AQ test and don’t have a diagnosis, LW could have a normal AS rate.)
various factors will affect who responds, but until you have a hypothesis about which way that bias points, all that does is add noise. I would predict that random sampling of people who ever look at the LW homepage would yield lower AS-spectrum results, but that random sampling of people who read regularly and/or comment probably wouldn’t.
It’s a convenience sample!
This is grossly overstated, because the survey respondents are in no way known to be representative of LessWrong. In fact, I would suspect a very strong selection bias. Before you can go saying anything about the LessWrong community, you need some response statistics. How many people were offered the survey? (You could get this from the server logs. Your survey wasn’t promoted and was only up for a day or so, an exposure that already subsets its audience from “LessWrong”) How many of the people who saw the survey decided to fill it out? How do the people who filled out the survey differ from the people who did not fill out the survey? I doubt the answers to any of these questions would be satisfying unless you started with a random sample, then individually and privately invited each of the sampled people to participate. Even then, the response bias might be bad enough to preclude any conclusions about LessWrong.
This is a good point, and we should consider it if we design a poll feature for LW, for example the poll could be made available only to registered users, and could track which of them saw it, and perhaps even give Karma rewards to those users who complete it to ensure a higher response rate.
Just out of interest, what’s your prediction of the sample mean AQ for those responses to my survey that are submitted after I put the note at the top of the post asking only previous non-responders to respond, assuming I get more than, say, 5?
Based mostly on Unnamed’s comment regarding a poll on homosexuality, I expect it would be lower than what you found for self-selected interested respondents, but I don’t have much confidence in this, and I’m not familiar enough with AQ variance to say how much lower.
I have some experience with trying to do survey research right (i.e. in a manner that results in true conclusions about the population), and I have seen so many ways that convenience samples can make me wrong (non-response bias is just a start), that I generally don’t bother much with trying to correct for one particular bias. Instead, I stick to random samples or use estimators that can handle convenience samples. But I am genuinely intrigued by your idea (from the very end of your post) to use a prior on non-response bias and a prior on AQ distribution to estimate the true AQ distribution of LW users. This wouldn’t handle e.g. problems with question wording or lies in responses, but I would expect those problems to be small in comparison to the non-response bias. Could you elaborate, or point me to any survey methods papers that explain this approach in detail? Where would you get the priors?
Yes, this is a good point.
Note that it’s still evidence, in the sense that changing the results of the survey ought to change your opinions about the AS-spectrum average over the cluster of Less Wrong readers.
Of course there is a potentially huge response bias—according to sitemeter there were 3000 visitors today, meaning that the response bias could in theory be up to 60 x . (which means that with maximum response bias, i.e. all nonresponders are scoring an average of 16 on the AQ test and don’t have a diagnosis, LW could have a normal AS rate.)
various factors will affect who responds, but until you have a hypothesis about which way that bias points, all that does is add noise. I would predict that random sampling of people who ever look at the LW homepage would yield lower AS-spectrum results, but that random sampling of people who read regularly and/or comment probably wouldn’t.
Wouldn’t the maximum response bias be if all the non-responders scored 0 on the scale?
This is highly implausible since we didn’t get a single response below 10, but yes, technically that would be an even bigger bias.