Basically, dude illustrates equivalence between p-values and Bayes factors and concludes that 17-25% of studies with a p-value acceptance threshold of 0.05 will be wrong. This implies that the lack of reproducibility in science isn’t necessarily due to egregious misconduct, etc., but rather insufficiently strict statistical standards.
So is this new/interesting, or do I just naively think so because it’s not my field?
Not a big deal. The estimate you’re impressed by can be done from power and prior odds like in Ioannides’s famous paper and are similar to Leek’s estimates from p-value distributions, and the recommendations baffle me—increase alpha?! P-value hacking is part of how we got here in the first place!
I don’t know any easy solutions to the low replication rate of many areas right now. It seems to be fundamentally a systematic problem of incentives. Even the easiest and most basic remedies like clinical trial registries are not being enforced, so it’s hopeless to expect reforms like making all studies well-powered. I do think that increasing alpha is unlikely to fix the problems and is likely to backfire by making things worse and rewarding cheaters & punishing honest researchers: the smaller the p-value required, the more you reward people who can run hundreds of analyses to get a p-value under the threshold and the more you punish honest researchers who did one analysis and stuck with it.
and concludes that 17-25% of studies with a p-value acceptance threshold of 0.05 will be wrong
That’s not what the dude concludes.
To quote the article itself (emphasis mine), “Although it is difficult to assess the proportion of all tested null hypotheses that are actually true, if one assumes that this proportion is approximately one-half, then these results
suggest that between 17% and 25% of marginally significant scientific findings are false.”
This seems like a big deal:
http://www.pnas.org/content/early/2013/10/28/1313476110.full.pdf
Basically, dude illustrates equivalence between p-values and Bayes factors and concludes that 17-25% of studies with a p-value acceptance threshold of 0.05 will be wrong. This implies that the lack of reproducibility in science isn’t necessarily due to egregious misconduct, etc., but rather insufficiently strict statistical standards.
So is this new/interesting, or do I just naively think so because it’s not my field?
Not a big deal. The estimate you’re impressed by can be done from power and prior odds like in Ioannides’s famous paper and are similar to Leek’s estimates from p-value distributions, and the recommendations baffle me—increase alpha?! P-value hacking is part of how we got here in the first place!
Is there a lower hanging fruit you have in mind?
I don’t know any easy solutions to the low replication rate of many areas right now. It seems to be fundamentally a systematic problem of incentives. Even the easiest and most basic remedies like clinical trial registries are not being enforced, so it’s hopeless to expect reforms like making all studies well-powered. I do think that increasing alpha is unlikely to fix the problems and is likely to backfire by making things worse and rewarding cheaters & punishing honest researchers: the smaller the p-value required, the more you reward people who can run hundreds of analyses to get a p-value under the threshold and the more you punish honest researchers who did one analysis and stuck with it.
That’s not what the dude concludes.
To quote the article itself (emphasis mine), “Although it is difficult to assess the proportion of all tested null hypotheses that are actually true, if one assumes that this proportion is approximately one-half, then these results suggest that between 17% and 25% of marginally significant scientific findings are false.”