I thought I’d seen that before on OB, but I’ve looked through, and while I’ve found a lot of stuff about being skeptical of science & medicine in particular, not that exact figure:
“Five out of six epidemiological studies have been contradicted in a very short period of time,” says Ioannidis, “while about one out of three randomized clinical trials were also refuted.”
Note that this is not 1⁄3 of all randomized studies, but just the highly cited ones, the ones that have any chance of an attempt at replication (and still 25% have no published attempt). It is not obvious to me whether these studies will be higher or lower quality than average. ETA: actually, there is a “control group” of articles published in the same prestigious journals, but with fewer citations. They are contradicted a little less often, p<.1, and not because people aren’t bothering to replicate them.
But there still quite a difference between claiming that 1⁄3 or 2⁄3 of randomized studies are wrong.
Yes, the 2⁄3 is probably an error on Eliezer’s part. He doesn’t say randomized, so he could be misremembering the 5⁄6, but the 5⁄6 is probably not the figure to quote and Ioannidis is a little misleading in putting it first. Only 6 of the 45 highly-cited studies were retrospective and I think that is representative of medical research in the prestigious journals. (the discussion of the control group should address this, but I don’t see it)
Thanks. Since I can’t find the original reference (if, indeed, it ever existed outside my imagination) these will do equally well to illustrate the point.
I thought I’d seen that before on OB, but I’ve looked through, and while I’ve found a lot of stuff about being skeptical of science & medicine in particular, not that exact figure:
http://www.overcomingbias.com/2009/07/clinical-trial-sloppiness.html
http://www.overcomingbias.com/2009/07/popular-fields-less-accurate.html
http://www.overcomingbias.com/2008/12/it-is-simply-no-longer-possible-to-believe.html
http://www.overcomingbias.com/2008/11/polisci-journal.html
http://www.overcomingbias.com/2007/08/anonymous-revie.html
http://www.overcomingbias.com/2007/01/conclusionblind.html
http://www.overcomingbias.com/2007/01/supping_with_th.html
http://www.overcomingbias.com/2006/12/academic_overco.html
I also remember seeing this on OB but was having trouble searching for some reason.
It’s ironic that you use a number for which you don’t know your source in an essay to recommend that public always release all their data.
Living by one’s own standards is always hard when you are a skeptic ;)
These seem relevant:
http://www.overcomingbias.com/2007/09/false-findings.html
“Why Most Published Research Findings Are False” I don’t think he gives a figure of 2⁄3, but he does say: “It can be proven that most claimed research findings are false.”
In fact, the origin of the statistic is an earlier paper by the same author, John PA Ioannidis, Contradicted and Initially Stronger Effects in Highly Cited Clinical Research. Journalistic coverage of the later paper quotes a description of the earlier:
Note that this is not 1⁄3 of all randomized studies, but just the highly cited ones, the ones that have any chance of an attempt at replication (and still 25% have no published attempt). It is not obvious to me whether these studies will be higher or lower quality than average. ETA: actually, there is a “control group” of articles published in the same prestigious journals, but with fewer citations. They are contradicted a little less often, p<.1, and not because people aren’t bothering to replicate them.
But there still quite a difference between claiming that 1⁄3 or 2⁄3 of randomized studies are wrong.
Yes, the 2⁄3 is probably an error on Eliezer’s part. He doesn’t say randomized, so he could be misremembering the 5⁄6, but the 5⁄6 is probably not the figure to quote and Ioannidis is a little misleading in putting it first. Only 6 of the 45 highly-cited studies were retrospective and I think that is representative of medical research in the prestigious journals. (the discussion of the control group should address this, but I don’t see it)
Thanks. Since I can’t find the original reference (if, indeed, it ever existed outside my imagination) these will do equally well to illustrate the point.