Overall, this indicates that response bias is probably not significantly skewing our picture of the LW audience...
As was mentioned in the comments to the previous post, the second round of the survey could have the same response bias as the first round, since you’re still only sampling from people who read a post about Aspergers and then chose to click through to a poll about Aspergers.
Imagine that out of every 10 people who regularly read LW, only 3 clicked through to the original article, and out of those 3, only 1 filled out the survey.
That’s two sequential filters. Now supposing that there’s a bias where people with high AQ scores are both more likely to click through to the article, and once they’re on the page more likely to bother to click through to the survey is reasonable. However, the data seems to rule that out.
The only hypothesis that still remains is that there is a big bias for clicking through to the article, but no bias at all for actually taking the (considerable) time to fill out the survey.
This seems somewhat contrived to me. Why would we expect that?
Also, we don’t know that there was “no bias at all” for taking the survey, just that the net change in bias between the first group and the second group was relatively small. I expected there to be a big bias in who made it through the first filter (reading the original article and finding out about the poll in its last paragraph) and multiple additional biases which would partially cancel out, and predicted that the net effect of these additional biases would be for the second group’s AQ to be a bit lower than the first group’s (but still substantially higher than the true population AQ). For instance, only some of the nonresponders from the first round actually saw your request for them to take the second-round survey, and they might tend to be high-AQ because they needed to look at your second Aspergers post to see the request.
I made this prediction about the AQ scores with low confidence, since it’s hard to guess the relative sizes of all of these potential biases, or even to identify every relevant bias. For instance, the data are showing that there were slightly more people who have been diagnosed with Aspergers in the second round than the first, and this could reflect a genuine difference (rather than random variation) caused by another bias: people who have been clinically tested for Aspergers might have been more likely to read the post but less likely to go on to take the survey, since they had less to learn from it.
My main point in this discussion is that, for future surveys, it’s better to try to avoid selection effects in the first place than to try to account for them after the fact, since they can introduce a lot of uncertainty which is hard to get rid of.
One way to think of it is that there were three filters:
only some LW visitors became aware of the survey when you first posted it (by reading about it in the last paragraph of your article)
of those who made it through filter 1, only some took the survey right then (in round 1)
of those who made it through filter 1 and did not make it through filter 2, only some took the survey later in round 2
With all three filters, it seemed like high AQ people would be more likely to make it through, but it was hard to estimate how strongly each filter would select for high AQ. I expected filter 1 to be the strongest (at selecting for high AQ), and filter 2 to be a bit stronger than filter 3. The data suggest that filter 3 was (if anything) very slightly stronger than filter 2, which requires some updating. But since one group went through filters 1 & 2 and the other went through filters 1 & 3, the data don’t speak directly to the strength of filter 1. You’re inferring that all three filters are probably relatively weak, but I don’t see a good reason to conclude that about filter 1.
As was mentioned in the comments to the previous post, the second round of the survey could have the same response bias as the first round, since you’re still only sampling from people who read a post about Aspergers and then chose to click through to a poll about Aspergers.
Imagine that out of every 10 people who regularly read LW, only 3 clicked through to the original article, and out of those 3, only 1 filled out the survey.
That’s two sequential filters. Now supposing that there’s a bias where people with high AQ scores are both more likely to click through to the article, and once they’re on the page more likely to bother to click through to the survey is reasonable. However, the data seems to rule that out.
The only hypothesis that still remains is that there is a big bias for clicking through to the article, but no bias at all for actually taking the (considerable) time to fill out the survey.
This seems somewhat contrived to me. Why would we expect that?
That is exactly what Psychohistorian expected.
Also, we don’t know that there was “no bias at all” for taking the survey, just that the net change in bias between the first group and the second group was relatively small. I expected there to be a big bias in who made it through the first filter (reading the original article and finding out about the poll in its last paragraph) and multiple additional biases which would partially cancel out, and predicted that the net effect of these additional biases would be for the second group’s AQ to be a bit lower than the first group’s (but still substantially higher than the true population AQ). For instance, only some of the nonresponders from the first round actually saw your request for them to take the second-round survey, and they might tend to be high-AQ because they needed to look at your second Aspergers post to see the request.
I made this prediction about the AQ scores with low confidence, since it’s hard to guess the relative sizes of all of these potential biases, or even to identify every relevant bias. For instance, the data are showing that there were slightly more people who have been diagnosed with Aspergers in the second round than the first, and this could reflect a genuine difference (rather than random variation) caused by another bias: people who have been clinically tested for Aspergers might have been more likely to read the post but less likely to go on to take the survey, since they had less to learn from it.
My main point in this discussion is that, for future surveys, it’s better to try to avoid selection effects in the first place than to try to account for them after the fact, since they can introduce a lot of uncertainty which is hard to get rid of.
Yeah, you actually predicted that the second responders would be slightly less AS-ish than the first responders, but actually they are slightly more.
To be honest, this result surprised me too, I expected that the AQ scores would go down, so I’m updating towards the “no large net bias” hypothesis.
One way to think of it is that there were three filters:
only some LW visitors became aware of the survey when you first posted it (by reading about it in the last paragraph of your article)
of those who made it through filter 1, only some took the survey right then (in round 1)
of those who made it through filter 1 and did not make it through filter 2, only some took the survey later in round 2
With all three filters, it seemed like high AQ people would be more likely to make it through, but it was hard to estimate how strongly each filter would select for high AQ. I expected filter 1 to be the strongest (at selecting for high AQ), and filter 2 to be a bit stronger than filter 3. The data suggest that filter 3 was (if anything) very slightly stronger than filter 2, which requires some updating. But since one group went through filters 1 & 2 and the other went through filters 1 & 3, the data don’t speak directly to the strength of filter 1. You’re inferring that all three filters are probably relatively weak, but I don’t see a good reason to conclude that about filter 1.