Only if it’s statistically significant. It could be a small enough effect that they don’t notice unless they’re looking for it (if you’re going to publish a finding from either extreme, you’re supposed to use a two-tailed test, so they’d presumably want something stronger than p = 0.05), but large enough to keep them from accidentally noticing the opposite effect.
Or alternately, it’s a large effect but the rarity of autism and of non-vaccinated kids makes it hard to reach statistical-significance given sampling error. So let’s see, the suggestion here is that the reason so few studies threw up a false positive was that the true effect was the opposite of the alternative, vaccines reduce autism.
Autism is… what, 0.5% of the general population of kids these days? And unvaccinated kids are, according to a random Mother Jones article, ~1.8%.
So let’s imagine that vaccines halve the risk of autism down from the true 1.0% to the observed 0.5% (halving certainly seems like a ‘large’ effect to me), autism has the true base rate of 1.0% in unvaccinated, and the unvaccinated make up 1.8% of the population. If we randomly sampled the population in general, how much would we have to sample in order to detect a difference in autism rates between the vaccinated & unvaccinated?
The regular R function I’d use for this, power.prop.test, doesn’t work since it assumes balanced sample sizes, not 1.8% in one group and 98.2% in the other. I could write a simulation to do the power calculation for a prop.test since the test itself handles imbalanced sample sizes, but then I googled and found someone had written something very similar for the Wilcoxon u-test, so hey, I’ll use the samplesize library instead; filling in the relevant values, we find for a decent chance of detecting such a correlation of vaccination with reduced autism, it takes:
a total n=90k. I’m guessing that most studies don’t get near that.
Of course, a lot of that penalty is going towards picking up enough kid who are both autistic and unvaccinated, so one could do better by trying to preferentially sample either of those groups, but then one gets into thorny questions about whether one’s convenience samples are representative and biased in some way...
As the original article says, if there was no effect, you’d expect a few studies to get p < 0.05 by chance. Similarly, if there was no effect, you’d expect a few studies to get p > 0.95 by chance, suggesting that vaccines prevent autism. If vaccines do prevent autism, then it would be even more likely to have p > 0.95.
Not all statistical analysis has to be preregistered. If a data has a trend that suggest vaccination might reduce autism I’m sure the researchers would run a test for it.
If the study is underpowered to find a effect in that direction it’s also like to be underpowered to find a effect in the other direction.
Can someone with more statistical expertise run a test to see whether the studies are underpowered to pick up effects in either direction?
Only if it’s statistically significant. It could be a small enough effect that they don’t notice unless they’re looking for it (if you’re going to publish a finding from either extreme, you’re supposed to use a two-tailed test, so they’d presumably want something stronger than p = 0.05), but large enough to keep them from accidentally noticing the opposite effect.
Or alternately, it’s a large effect but the rarity of autism and of non-vaccinated kids makes it hard to reach statistical-significance given sampling error. So let’s see, the suggestion here is that the reason so few studies threw up a false positive was that the true effect was the opposite of the alternative, vaccines reduce autism.
Autism is… what, 0.5% of the general population of kids these days? And unvaccinated kids are, according to a random Mother Jones article, ~1.8%.
So let’s imagine that vaccines halve the risk of autism down from the true 1.0% to the observed 0.5% (halving certainly seems like a ‘large’ effect to me), autism has the true base rate of 1.0% in unvaccinated, and the unvaccinated make up 1.8% of the population. If we randomly sampled the population in general, how much would we have to sample in order to detect a difference in autism rates between the vaccinated & unvaccinated?
The regular R function I’d use for this,
power.prop.test, doesn’t work since it assumes balanced sample sizes, not 1.8% in one group and 98.2% in the other. I could write a simulation to do the power calculation for aprop.testsince the test itself handles imbalanced sample sizes, but then I googled and found someone had written something very similar for the Wilcoxon u-test, so hey, I’ll use the samplesize library instead; filling in the relevant values, we find for a decent chance of detecting such a correlation of vaccination with reduced autism, it takes:a total n=90k. I’m guessing that most studies don’t get near that.
Of course, a lot of that penalty is going towards picking up enough kid who are both autistic and unvaccinated, so one could do better by trying to preferentially sample either of those groups, but then one gets into thorny questions about whether one’s convenience samples are representative and biased in some way...
As the original article says, if there was no effect, you’d expect a few studies to get p < 0.05 by chance. Similarly, if there was no effect, you’d expect a few studies to get p > 0.95 by chance, suggesting that vaccines prevent autism. If vaccines do prevent autism, then it would be even more likely to have p > 0.95.
Not all statistical analysis has to be preregistered. If a data has a trend that suggest vaccination might reduce autism I’m sure the researchers would run a test for it.
If the study is underpowered to find a effect in that direction it’s also like to be underpowered to find a effect in the other direction.
Can someone with more statistical expertise run a test to see whether the studies are underpowered to pick up effects in either direction?