No. Your excerpt was a poor exposition of the standard precautionary principle I regard as entirely useless.
Reading your link now, I have even less reason to pay attention to it. It’s a random press release about a presumably small unreplicated animal study in a species I don’t know to be particularly germane to humans (eg. chimps) about changes of unclear importance in a body system with no human analogues (“In birds, switching off GnRH causes the gonads—testes and ovary—to shrink as part of the birds’ yearly cycle.”) with doses potentially high enough to be completely irrelevant to human supplementation (injecting melatonin?). I haven’t even read the study!
Combine all the conditionals here (the smallness and lack of replication alone knocks down the chance this means anything about anything to well under 50%), and I don’t see why I would update at all (not being an AI or anything which can represent degrees of belief with 64-bit floats).
If anything, I think this sort of study is a good example of why animal studies should be ignored in discussing supplements.
On the general topic of animal model external validity & translation to humans (with obvious relevance to supplements & nootropics), here are the major systematic reviews, meta-analyses, and articles criticizing the routine failures of animal models to provide any meaningful information about dangers or benefits in humans, and documenting the even lower quality of animal experiments than usual in (human) medicine or psychology:
I spent the day reading up on the topic. The transfer or replication rates range from around 40% on the upper end to zero in some cases involving as many as hundreds of attempts to transfer. The methodological quality of the animal studies are usually terrible with hardly any blinding and randomization rare (and when it is done, one author indicates that researchers surveyed would say it was done as informally as grabbing random mice/rats out of the cages), and the publication biases at play seem to be even larger than in human studies.
(Reading the papers, I found myself disturbed by the ethical implications: when human studies fall to publication bias, ‘all’ that does is waste many millions of dollars and impede the progress of science and substantially inconvenience many subjects and put them at risk; but when at least a third of all animal experiments never get written up, and half of chimpanzee medical/biological studies never get cited when they do get published, that means that the billions of animals gone through every few years are mutilated and tortured and killed for absolutely nothing at all. This makes me much more sympathetic to the NIH’s recent retirement of its research chimpanzees.)
There are some interesting examples, though; from one of the links:
In animal models of acute ischemic stroke, about 500 “neuroprotective” treatment strategies have been reported to improve outcome, but only aspirin and very early intravenous thrombolysis with alteplase (recombinant tissue-plasminogen activator) have proved effective in patients, despite numerous clinical trials of other treatment strategies [8],[9].
From the toxicology paper:
“Karnofsky’s Law, which states ‘Anything can be teratogenic if given in the right dose, to the right species, at the right time’. If every single drug, chemical and indeed substance can be teratogenic in some particular animal at some specific dose, then to produce a positive result one need only find a suitably sensitive species and administer a suitably high dose (Scialli, 1992). This may well explain the inclusion of various everyday substances, some of which are intrinsic components of the mother’s body and/or the developing conceptus and which are essential for life itself, in the list of teratogens in ’Dangerous Properties of Industrial Materials@ (Lewis, 1989). These include drinking water (Turbow et al., 1971) and table salt (Nishimura and Miyamoto, 1969); oxygen; sugars in the form of sucrose and lactose; palm oil, corn oil and nutmeg oil; other naturally occurring food constituents such as cholesterol and papain (prevalent in pineapples); vitamins such as A, D2, K, B7 and B12 (the B vitamins are frequently found in pregnancy supplements); naturally occurring and essential hormones such as estradiol, progesterone and various prostaglandins; the amino acid methionine, and the DNA constituent adenine.”
Animal models such as mice can simply be irrelevant to humans, leading to cases like <150 sepsis clinical trials all failing—because the drugs worked in mice but humans have a completely different set of genetic reactions to inflammation.
(I imagine that the latter is particularly relevant; how many thousands of mouse studies on inflammation were part of the evidence base for those <150 clinical trials? Probably quite a few. And it seems that they all are essentially irrelevant to anything in humans. Now, imagine the translation rate for bird to primate, based on a bird system which doesn’t even exist in humans...)
Thanks for doing all this research! You should make a page on gwern.net, you’re wasting your talents going in to this kind of depth in an ancient comments thread ;)
There are some interesting examples, though; from the second link:
In animal models of acute ischemic stroke, about 500 “neuroprotective” treatment strategies have been reported to improve outcome, but only aspirin and very early intravenous thrombolysis with alteplase (recombinant tissue-plasminogen activator) have proved effective in patients, despite numerous clinical trials of other treatment strategies [8],[9].
Animal models such as mice can simply be irrelevant to humans, leading to cases like <150 sepsis clinical trials all failing—because the drugs worked in mice but humans have a completely different set of genetic reactions to inflammation.
(I imagine that the latter is particularly relevant; how many thousands of mouse studies on inflammation were part of the evidence base for those <150 clinical trials? Probably quite a few. And it seems that they all are essentially irrelevant to anything in humans. Now, imagine the translation rate for bird to primate, based on a bird system which doesn’t even exist in humans...)
I suspect it might also depend, among other things, on how closely related those animals are to us. I would likely take your study more seriously if it was done on mammals, and even more so if it was done on primates.
I’m confused… did you update at all on the possibility of harms from long-term melatonin use after reading the article?
No. Your excerpt was a poor exposition of the standard precautionary principle I regard as entirely useless.
Reading your link now, I have even less reason to pay attention to it. It’s a random press release about a presumably small unreplicated animal study in a species I don’t know to be particularly germane to humans (eg. chimps) about changes of unclear importance in a body system with no human analogues (“In birds, switching off GnRH causes the gonads—testes and ovary—to shrink as part of the birds’ yearly cycle.”) with doses potentially high enough to be completely irrelevant to human supplementation (injecting melatonin?). I haven’t even read the study!
Combine all the conditionals here (the smallness and lack of replication alone knocks down the chance this means anything about anything to well under 50%), and I don’t see why I would update at all (not being an AI or anything which can represent degrees of belief with 64-bit floats).
If anything, I think this sort of study is a good example of why animal studies should be ignored in discussing supplements.
Good points.
Have there been any meta-analyses of how well supplement studies on animals tend to transfer to humans?
On the general topic of animal model external validity & translation to humans (with obvious relevance to supplements & nootropics), here are the major systematic reviews, meta-analyses, and articles criticizing the routine failures of animal models to provide any meaningful information about dangers or benefits in humans, and documenting the even lower quality of animal experiments than usual in (human) medicine or psychology:
“Nimodipine in animal model experiments of focal cerebral ischemia: a systematic review”, Horn 2001; review
“Wound healing in cell studies and animal model experiments by Low Level Laser Therapy; were clinical studies justified? A systematic review”, Lucas et al 2002; meta-analysis
“Does animal experimentation inform human healthcare? Observations from a systematic review of international animal experiments on fluid resuscitation”, Roberts et al 2002; meta-analysis
“Systematic reviews of animal experiments”, Sandercock & Roberts 2002
“Meta-analysis of the effects of endothelin receptor blockade on survival in experimental heart failure”, Lee et al 2003; meta-analysis
“Where is the evidence that animal research benefits humans?”, Pound et al 2004; review
“The use of animal models in the study of complex disease: all else is never equal or why do so many human studies fail to replicate animal findings?”, Williams et al 2004; essay
“The future of teratology research is in vitro”, Bailey et al 2005; review
“How good are rodent models of carcinogenesis in predicting efficacy in humans? A systematic review and meta-analysis of colon chemoprevention in rats, mice and men”, Corpet & Pierre 2005; meta-analysis
“Surveying the literature from animal experiments”, Lemon & Dunnett 2005; essay
“Systematic review and meta-analysis of the efficacy of FK506 in experimental stroke”, Macleod et al 2005; meta-analysis
“Systematic review and meta-analysis of the efficacy of melatonin in experimental stroke”, Macleod et al 2005; meta-analysis
“Methodological quality of animal studies on neuroprotection in focal cerebral ischaemia”, van der Worp et al 2005; review
“Nitric oxide synthase inhibitors in experimental ischemic stroke and their effects on infarct size and cerebral blood flow: a systematic review”, Willmot et al 2005; meta-analysis
“A systematic review of nitric oxide donors and L-arginine in experimental stroke; effects on infarct size and cerebral blood flow”, Willmot et al 2005; meta-analysis
“Translation of Research Evidence From Animals to Humans”, Hackam 2006; review “1,026 experimental treatments in acute stroke”, O’Collins et al 2006; review
“A Systematic Review of Systematic Reviews and Meta-Analyses of Animal Experiments with Guidelines for Reporting”, Peters 2006; review
“Translating animal research into clinical benefit”, Hackam 2007; essay
“Systematic Reviews of Animal Experiments Demonstrate Poor Human Clinical and Toxicological Utility”%20641-659.pdf), Knight 2007; review
“Comparison of treatment effects between animal experiments and clinical trials: systematic review”, Perel et al 2007; review
“How can we improve the pre-clinical development of drugs for stroke?”, Sena et al 2007; essay
“Systematic Reviews of Animal Experiments Demonstrate Poor Contributions to Human Healthcare”%2089-96.pdf), Knight 2008; essay
“Evidence-Based Data From Animal and Human Experimental Studies on Pain Relief With Antidepressants: A Structured Review”, Fishbain et al 2008; review
“Are animal models as good as we think?”, Wall & Shani 2008; essay
“Publication bias in reports of animal stroke studies leads to major overstatement of efficacy”, Sena et al 2010; meta-analysis
“Can Animal Models of Disease Reliably Inform Human Studies?”, Worp 2010; essay
“Human relevance of pre-clinical studies in stem cell therapy: systematic review and meta-analysis of large animal models of ischaemic heart disease”, van der Spoel et al 2011; meta-analysis
I spent the day reading up on the topic. The transfer or replication rates range from around 40% on the upper end to zero in some cases involving as many as hundreds of attempts to transfer. The methodological quality of the animal studies are usually terrible with hardly any blinding and randomization rare (and when it is done, one author indicates that researchers surveyed would say it was done as informally as grabbing random mice/rats out of the cages), and the publication biases at play seem to be even larger than in human studies.
(Reading the papers, I found myself disturbed by the ethical implications: when human studies fall to publication bias, ‘all’ that does is waste many millions of dollars and impede the progress of science and substantially inconvenience many subjects and put them at risk; but when at least a third of all animal experiments never get written up, and half of chimpanzee medical/biological studies never get cited when they do get published, that means that the billions of animals gone through every few years are mutilated and tortured and killed for absolutely nothing at all. This makes me much more sympathetic to the NIH’s recent retirement of its research chimpanzees.)
There are some interesting examples, though; from one of the links:
From the toxicology paper:
From my appendix:
(I imagine that the latter is particularly relevant; how many thousands of mouse studies on inflammation were part of the evidence base for those <150 clinical trials? Probably quite a few. And it seems that they all are essentially irrelevant to anything in humans. Now, imagine the translation rate for bird to primate, based on a bird system which doesn’t even exist in humans...)
Thanks for doing all this research! You should make a page on gwern.net, you’re wasting your talents going in to this kind of depth in an ancient comments thread ;)
Perish the thought of there being any waste!
I don’t know of any good meta-analysis or review, aside from some quick searching turning up “Translation of Research Evidence From Animals to Humans” & “Can Animal Models of Disease Reliably Inform Human Studies?” which seems to be narrower than one would want (examining treatment for specific issues in animal models translating to successful treatment in humans, which is much more likely to translate than your press release).
There are some interesting examples, though; from the second link:
From my appendix:
(I imagine that the latter is particularly relevant; how many thousands of mouse studies on inflammation were part of the evidence base for those <150 clinical trials? Probably quite a few. And it seems that they all are essentially irrelevant to anything in humans. Now, imagine the translation rate for bird to primate, based on a bird system which doesn’t even exist in humans...)
I suspect it might also depend, among other things, on how closely related those animals are to us. I would likely take your study more seriously if it was done on mammals, and even more so if it was done on primates.