The bigger response bias problem comes from the fact that the survey was presented as a survey about Asperger’s, which means that people who are on the autism spectrum probably tended to be more interested in it and more likely to respond to the survey. For instance, a survey about sexual orientation at Cognitive Daily found that rates of homosexuality increased from about 5% of respondents when the sexuality question was tacked on to an unrelated survey to about 15% when the poll was advertised as being about sexual orientation.
Perhaps LW should develop a convention for having surveys where the purpose of the survey isn’t explained until after people have taken it.
OK, so if take the homosexuality survey as a data point, we can say that selection bias in a survey can increase the reported rate of a behavior by 3 (5% homosexual in normal population, 15% when the survey is about sexual orientation).
If we assume the same increased rate applies in this case, we can divide 5% by three and get 1.666%, still far above the normal 0.36% of the general population.
The ratio could easily be higher than 3:1. The Cognitive Daily survey had a one-question poll right there in the top post on the site’s home page, while this survey required following a link at the bottom of a long post about Asperger’s. That means that this sample was probably more filtered for those who are most motivated and interested in the topic, which will tend to make the sample more biased.
Somewhere around 1% of the people who visited Less Wrong took the poll (82 respondents when there are about 8,000 visitors/day), and it’s not out of the realm of possibility for the response rate among people with Asperger’s to have been 14%, which would be enough to account for the 14:1 ratio between the 5% rate of Asperger’s in the sample and the 0.36% rate in the general population.
It’s hard to estimate the size of a selection effect, especially when the response rate is so low, which makes it hard to undo the bias after the fact to figure out the actual rate. That’s why I’d recommend putting some convention in place for surveys that avoid selection effects as much as possible, if we’re going to try to use surveys to draw conclusions about the LW community.
Also, wanting to take a multiple choice test that outputs a numeric score is going to have a higher correlation with autism/Asperger’s than with homosexuality :)
Just out of interest, what’s your prediction of the sample mean AQ for those responses to my survey that are submitted after I put the note at the top of the post asking only previous non-responders to respond, assuming I get more than, say, 5?
It’s hard to say, since the new set of respondents could have the same selection bias. They would be people who 1) opened your “Do you have High-Functioning Asperger’s Syndrome?” post, 2) read the last paragraph of the post which mentioned the poll, 3) considered taking the poll but didn’t, 4) opened this post about the Asperger’s poll (or re-opened your previous post) and saw your request, and 5) decided to comply with your request and take the poll. Steps 1, 2, and 4 all seem likely to select for high-AQ people; steps 3 & 5 might shift your sample in the opposite direction but probably not as strongly (although the explicit concerns about nonresponse bias could make low-AQ people especially motivated to take the survey to counteract that bias).
To answer your question, I’ll guess that the new mean will be slightly lower than the first result but not by much. Let’s say a mean of 25 for your new sample, and a mean of 21 if we ever give this survey to a representative sample of the LW community (or at least a sample that doesn’t know that the survey is about Asperger’s).
The bigger response bias problem comes from the fact that the survey was presented as a survey about Asperger’s, which means that people who are on the autism spectrum probably tended to be more interested in it and more likely to respond to the survey. For instance, a survey about sexual orientation at Cognitive Daily found that rates of homosexuality increased from about 5% of respondents when the sexuality question was tacked on to an unrelated survey to about 15% when the poll was advertised as being about sexual orientation.
Perhaps LW should develop a convention for having surveys where the purpose of the survey isn’t explained until after people have taken it.
OK, so if take the homosexuality survey as a data point, we can say that selection bias in a survey can increase the reported rate of a behavior by 3 (5% homosexual in normal population, 15% when the survey is about sexual orientation).
If we assume the same increased rate applies in this case, we can divide 5% by three and get 1.666%, still far above the normal 0.36% of the general population.
The ratio could easily be higher than 3:1. The Cognitive Daily survey had a one-question poll right there in the top post on the site’s home page, while this survey required following a link at the bottom of a long post about Asperger’s. That means that this sample was probably more filtered for those who are most motivated and interested in the topic, which will tend to make the sample more biased.
Somewhere around 1% of the people who visited Less Wrong took the poll (82 respondents when there are about 8,000 visitors/day), and it’s not out of the realm of possibility for the response rate among people with Asperger’s to have been 14%, which would be enough to account for the 14:1 ratio between the 5% rate of Asperger’s in the sample and the 0.36% rate in the general population.
It’s hard to estimate the size of a selection effect, especially when the response rate is so low, which makes it hard to undo the bias after the fact to figure out the actual rate. That’s why I’d recommend putting some convention in place for surveys that avoid selection effects as much as possible, if we’re going to try to use surveys to draw conclusions about the LW community.
Also, wanting to take a multiple choice test that outputs a numeric score is going to have a higher correlation with autism/Asperger’s than with homosexuality :)
Just out of interest, what’s your prediction of the sample mean AQ for those responses to my survey that are submitted after I put the note at the top of the post asking only previous non-responders to respond, assuming I get more than, say, 5?
It’s hard to say, since the new set of respondents could have the same selection bias. They would be people who 1) opened your “Do you have High-Functioning Asperger’s Syndrome?” post, 2) read the last paragraph of the post which mentioned the poll, 3) considered taking the poll but didn’t, 4) opened this post about the Asperger’s poll (or re-opened your previous post) and saw your request, and 5) decided to comply with your request and take the poll. Steps 1, 2, and 4 all seem likely to select for high-AQ people; steps 3 & 5 might shift your sample in the opposite direction but probably not as strongly (although the explicit concerns about nonresponse bias could make low-AQ people especially motivated to take the survey to counteract that bias).
To answer your question, I’ll guess that the new mean will be slightly lower than the first result but not by much. Let’s say a mean of 25 for your new sample, and a mean of 21 if we ever give this survey to a representative sample of the LW community (or at least a sample that doesn’t know that the survey is about Asperger’s).
The results of the new survey will, I am sure, be enlightening!