Dealing with the high quantity of scientific error in medicine
In a recent article, John Ioannidis describes a very high proportion of medical research as wrong.
Still, Ioannidis anticipated that the community might shrug off his findings: sure, a lot of dubious research makes it into journals, but we researchers and physicians know to ignore it and focus on the good stuff, so what’s the big deal? The other paper headed off that claim. He zoomed in on 49 of the most highly regarded research findings in medicine over the previous 13 years, as judged by the science community’s two standard measures: the papers had appeared in the journals most widely cited in research articles, and the 49 articles themselves were the most widely cited articles in these journals. These were articles that helped lead to the widespread popularity of treatments such as the use of hormone-replacement therapy for menopausal women, vitamin E to reduce the risk of heart disease, coronary stents to ward off heart attacks, and daily low-dose aspirin to control blood pressure and prevent heart attacks and strokes. Ioannidis was putting his contentions to the test not against run-of-the-mill research, or even merely well-accepted research, but against the absolute tip of the research pyramid. Of the 49 articles, 45 claimed to have uncovered effective interventions. Thirty-four of these claims had been retested, and 14 of these, or 41 percent, had been convincingly shown to be wrong or significantly exaggerated. If between a third and a half of the most acclaimed research in medicine was proving untrustworthy, the scope and impact of the problem were undeniable. That article was published in the Journal of the American Medical Association.
Part of the problem is that surprising results get more interest, and surprising results are more likely to be wrong. (I’m not dead certain of this—if the baseline beliefs are highly likely to be wrong, surprising beliefs become somewhat less likely to be wrong.) Replication is boring. Failure to replicate a bright shiny surprising belief is boring. A tremendous amount isn’t checked, and that’s before you start considering that a lot of medical research is funded by companies that want to sell something.
Corollary 1: The smaller the studies conducted in a scientific field, the less likely the research findings are to be true.
Corollary 2: The smaller the effect sizes in a scientific field, the less likely the research findings are to be true.
Corollary 3: The greater the number and the lesser the selection of tested relationships in a scientific field, the less likely the research findings are to be true.
Corollary 4: The greater the flexibility in designs, definitions, outcomes, and analytical modes in a scientific field, the less likely the research findings are to be true.
Corollary 5: The greater the financial and other interests and prejudices in a scientific field, the less likely the research findings are to be true.
Corollary 6: The hotter a scientific field (with more scientific teams involved), the less likely the research findings are to be true.
The culture at LW shows a lot of reliance on small inferential psychological studies—for example that doing a good deed leads to worse behavior later. Please watch out for that.
A smidgen of good news: Failure to Replicate, a website about failures to replicate psychological findings. I think this could be very valuable, and if you agree, please boost the signal by posting it elsewhere.
From Failure to Replicate’s author—A problem with metastudies:
Eventually, someone else comes across this small literature and notices that it contains “mixed findings”, with some studies finding an effect, and others finding no effect. So this special someone–let’s call them the Master of the Gnomes–decides to do a formal meta-analysis. (A meta-analysis is basically just a fancy way of taking a bunch of other people’s studies, throwing them in a blender, and pouring out the resulting soup into a publication of your very own.) Now you can see why the failure to publish null results is going to be problematic: What the Master of the Gnomes doesn’t know about, the Master of the Gnomes can’t publish about. So any resulting meta-analytic estimate of the association between lawn gnomes and subjective well-being is going to be biased in the positive direction. That is, there’s a good chance that the meta-analysis will end up saying lawn gnomes make people very happy,when in reality lawn gnomes only make people a little happy, or don’t make people happy at all.
The people I’ve read who gave advice based on Ioannidis article strongly recommended eating paleo. I don’t think this is awful advice in the sense that a number of people seem to actually feel better following it, and I haven’t heard of disasters resulting from eating paleo. However, I don’t know that it’s a general solution to the problems of living with a medical system which does necessary work some of the time, but also is wildly inaccurate and sometimes destructive.
The following advice is has a pure base of anecdote, but at least I’ve heard a lot of them from people with ongoing medical problems. (Double meaning intended.)
Before you use prescription drugs and/or medical procedures, make sure there’s something wrong with you. Keep an eye out for side effects and the results of combined medicines. Check for evidence that whatever you’re thinking about doing actually helps. Be careful with statins—they can cause reversible memory problems and permanent muscle weakness. Choose a doctor who listens to you.
Forum about self-experimentation—note: even Seth Roberts is apt to oversell his results as applying to everyone.
Link about the failure to replicate site found here.
- Follow-up on ESP study: “We don’t publish replications” by (12 Jul 2011 20:48 UTC; 112 points)
- Using degrees of freedom to change the past for fun and profit by (7 Mar 2012 2:51 UTC; 65 points)
- [Link] An argument for Low-hanging fruit in Medicine by (22 Feb 2012 15:43 UTC; 16 points)
- 's comment on Cold fusion: real after all? by (18 Apr 2013 20:09 UTC; 10 points)
- 's comment on HELP: How do minimum wage laws harm people? by (26 Oct 2010 11:33 UTC; 7 points)
- Informed consent bias in RCTs? by (27 Jan 2012 2:31 UTC; 3 points)
Okay, but don’t make the mistake of the guy who says “The mainstream media is all lies—so I’ll only trust what I read on shady Internet conspiracy sites”. Saying that there are likely flaws in mainstream medical research doesn’t license you to discount any specific medical finding unless you have particular reason to believe that finding is false. And it certainly doesn’t license you to place more credibility in small, poorly performed studies that contradict large, well-performed studies, or in fringe theories that contradict mainstream theories. Unless you hold your favorite theory, be it anti-vax, paleo-diet, or whatever, to the same high standard you hold the medical mainstream, every true fact you learn about flaws in medical research makes you stupider.
The study mentioned above looks at exciting cutting-edge research over the past decade. It says that 40% or so was proven wrong. This is good and to the credit of medical science! It means the system is working as it should in retesting things and getting the false stuff out. The basis of science isn’t getting everything right the first time, it’s making sure everyone’s work gets checked and double-checked until only the truth survives. An unreplicated study in almost any area is an intriguing possibility and nothing more; medicine is no exception. If the media makes a big deal about a new study and publishes “VITAMIN B CURES BREAST CANCER!!!” in 72 point font in the newspapers, that is an interesting fact about the media and the people who believe it, but not an interesting fact about medical science.
Good doctors are both conservative and utilitarian. They stick to older, well-proven treatments unless the advantage of a new treatment is so great that it outweighs the uncertainty and risks involved. IMHO the medical consensus has been right on the important things a surprising amount of the time.
I would strongly discourage people from bewaring statins overly much. I don’t see anything by Ioannidis saying the studies surrounding statins are particularly bad. Ioannidis says research is less likely to be true if it has low sample sizes, low effect sizes, bias, and a wide net. There have been several statin trials with sample sizes in the thousands to tens of thousands (see: JUPITER, SSSS, etc.) They’ve found that death rate from heart attacks in people correctly prescribed statin goes down by 30%, which is not at all a small effect size. Many such trials have not been linked to statin manufacturers or anyone with an axe to grind. And because people already know statins are supposed to reduce cholesterol, there is much less of a wide net than if you were to give a bunch of people statins and, say, see if any diseases became less common—the studies had a clearly designated endpoint, which they achieved.
Are there people who suggest the side effects of statins are worse than everyone else thinks? A few, and based off of very little evidence (I believe the idea that statins cause memory dysfunction is based mostly off isolated case reports, and there are only 60 out of many years of hundreds of thousands of people on statins—basically background noise). I haven’t investigated this thoroughly, but the side effects would have to be pretty darned bad and pretty darned robust to stop prescribing a drug with an NNT in the two digits (ie it takes under 100 statin prescriptions to prevent one heart attack), and I treat people trying to exaggerate drug side effects as just as real a failure mode as doctors trying to exaggerate drug benefits, and use just as much caution.
The advice in the third-to-last paragraph, except perhaps the specific singling out of statins, remains excellent.
Overall, the use of the term “license” here raises yellow flags for me (see Hero Licensing for the basic reason). It conflates social standing with epistemic standing. The first paragraph here seems a bit confused in other ways too, let me try to break it up into what I see as comparatively crisp distinct claims.
CLAIM: Saying that there are likely flaws in mainstream medical research [...] doesn’t license you to place more credibility in small, poorly performed studies that contradict large, well-performed studies, or in fringe theories that contradict mainstream theories.
This seems basically true, and is an application of Beware Isolated Demands for Rigor.
CLAIM: Saying that there are likely flaws in mainstream medical research doesn’t license you to discount any specific medical finding unless you have particular reason to believe that finding is false.
The burden of evidence is on the people making claims that constrain our anticipations. We can and should discount all statistical findings in proportion to the evidence that they’re unreliable.
CLAIM: Unless you hold your favorite theory, be it anti-vax, paleo-diet, or whatever, to the same high standard you hold the medical mainstream, every true fact you learn about flaws in medical research makes you stupider.
Statistics isn’t the only sort of evidence. “The same high standards” isn’t necessarily meaningful here, as different kinds of standards are required for different kinds of evidence. There’s also checking a claim against your model of how the world works, and trying things to see whether they produce the claimed effect. The Paleo argument matches my underlying understanding of how the world works, while I’d find the opposite claim pretty counterintuitive. Then I tried eating vaguely Paleo and I felt noticeably better and lost 20 pounds. I have a bunch of friends who report vaguely similar results. Anecdotes like this don’t overrule strong statistical evidence, but that doesn’t mean much when there’s not strong statistical evidence. A little evidence is often better than none, and “standards” may not be the right paradigm here.
It’s nice to be careful not to overgeneralize from personal experience, and it’s also nice to be careful not to overgeneralize from unreplicated underpowered studies, especially when they find something counterintuitive that contradicts my life experience or would be hard for me to check.
That’s a very good point—medical science being bad does not imply that your favorite contrarian idea is good.
So if I understand you right, the moral that we should draw is that the most reliable treatments are the old ones. A new study constitutes very weak evidence in favor of anything.
I remain skeptical of medicine, but there is a particular kind of emotional/intellectual scam that is practiced by some alternative medicine practitioners, akin to what Yvain cautions.
Particularly when the standard, tested regimen (say chemo + radiation for cancer) has miserable side effects, it’s tempting to go instead for the herbal remedy (or the like).
But until those alternative treatments are subjected to large trials, what we’re trading, basically, is the flawed, but at least somewhat familiar and certainly broad-based approach of medical research for the much murkier world of individual “authorities” and anecdotes. It’s darkly ironic that the same folks who urge me to discount the authority of the medical establishment are eager to have me listen to their “authorities” instead.
The emotional (and generally unacknowledged) part of this scam is that the non-traditional recipe is particularly appealing when the standard remedy is horrible.
You may be right about the statins—part of what spooked me about them was running into a woman whose husband had taken permanent muscle damage from them, which suggested to me that the side effect might not be all that rare.
You mentioned that it was important for them to be correctly prescribed. How common is it for them to not be correctly prescribed?
Rhabdomyolysis, which I think is the kind of severe permanent muscle damage you’re talking about, is well-known enough as a side effect of statins that it’s taught in first year medical school classes. There was one statin that may have had a relatively high (1/2,000 per year) rhabdomyolysis rate and was withdrawn from the market after a couple of years for that reason. The statins currently on the market have about a 1⁄20,000/year rhabdomyolysis rate, which is actually low enough that no one is entirely sure it’s not background noise although no one’s taking any chances. Since they also have a 1+/500/year heart attack prevention rate, they prevent something like 50 heart attacks for each case of rhabdomyolysis they cause, which seems “worth it”.
Muscle damage rates increase by a lot if you take statins with fibrates (another cholesterol lowering drug). I think (not sure) that prescribing these two drugs together is Officially Discouraged, although there might be some leeway in cases of people with crazy high cholesterol. I’ve also heard having grapefruit juice with statins increases the risk (grapefruit juice messes with liver enzymes) but I’m not sure if that is practically important or just random clinical trivia.
As for correct prescription: I am only a student, I haven’t checked the official guidelines, and if you hear otherwise from any doctor trust the doctor and not me—however, as I understand it there is pretty good evidence for giving a statin to people who have already had a cardiovascular event in order to prevent a second one, and much weaker evidence (depending on whose studies and meta-analyses you prefer) for giving it to someone who’s never had a cardiovascular event. Many doctors give them to the latter category anyway just because irreversible side effects are so rare and they would rather be safe than sorry regarding heart attacks; I see some merit in both sides of the argument.
Thank you for doing the research.
I’ve had a little more time to think—how sure are you that the studies you cite were well-constructed?
My only evidence is that they’re mostly by large and respected institutions, considered exemplary by the medical community, and that reading a one page summary of them I didn’t come across anything that made me think they weren’t.
Muscle/nerve damage and amnesia as side effects of statins It sounds as though memory problems aren’t routinely monitored, and there’s no way of telling (or at least nothing that gets used) if there’s muscle damage until the patient gets into trouble.
This doesn’t prove that statins aren’t worth the risks.
I read David Freedman’s book Wrong a while ago. The article (which I have not read yet) appears to be a condensation of the book. I highlighted a few passages while I read. Here they are.
Quotes about the problem of groupthink from the chapter The Idiocy of Crowds:
There is almost no replication in science, and even when there is, the replication is ignored:
Scientists have a substantial motive to fudge research or commit outright fraud:
Whistle-blowing is strongly discouraged:
Publication bias, with the help of incompetence or by itself, can produce the same effect as fraud:
Peer review has much less value in selecting valid science than is popularly believed:
By capturing the process of peer review, bad scientists can suppress refutation of their own results:
In re the chilling effects of status: In Gawande’s The Checklist Manifesto, he says that one of the valuable effects of checklists is that they enable low status people (like nurses) to tell their “superiors” that some crucial step has been skipped.
Sorry to nitpick on typos, but the researcher’s name is spelled in four different ways in the post: Ionnides, Ioannidis, Ioannidde and Ioannides. I suppose not all can be correct.
I think it’s straightened out now—the second one was correct. Thanks.
Ok, that’s really funny :-)
I first read Ioannidis’ paper several months ago and enjoyed it immensely. The Atlantic article is also good and I hope that it gets some much-needed conversation energized.
A couple of remarks.
I do not like the title. Ioannidis has shown that most published medical research is wrong and the title of his paper why most published research findings are false is misleading to me. Many of the pitfalls which he describes are specific to the limits of experimenting on human subjects which is not transferable across the entire universe of experimental science. Some of the pitfalls are general and all scientists should pay attention to them; it would have been a smoother rhetorical move to reflect this in his title.
The biggest issues amongst my friends and co-workers are diet and neurological nutritional supplements. My experience is that both of these areas are currently a quicksand swamp where it seems like a study of some sort can be found to support many positions that are in contradiction; the most obvious example is the thing with low fat diets, Atkins diets, paleo diets, and the list is far too long to include them all. I have a book Sports Nutrition published by The American Dietetic Association, which is a compilation from Track people, Football people, Swimming people, &c—there are fifty separate articles. The advice is all over the map. The most interesting thing in that book was the people who said that protein powders are useless. I have my own ideas on diet but I find my social life is less complicated if I keep them to myself.
There is some interesting information and (some complete nonsense) in the book by Doctor Daniel Amen Making a good brain great. It may just be blowback to his enterprising attitude, but Amen attracted enough detractor attention to get himself an entry on the quackwatch web page. He supplies medical journal citations for nearly all the claims in his book. Some of his recommended supplements I had heard of before, such as ginko biloba. Some of his recommended supplements I had never heard of, such as coenzyme Q. Amen takes twenty-five brain supplements daily. It was reading his book, and seeing his claims, and seeing his citations, which convinced me of the Ioannidis claim long before I read the Ioannidis paper. It seems impossible that all of the Amen claims and citations could be valid; it seems far more likely that most of the Amen claims and citations are false.
There is one fascinating item in Amen’s book which I will share as an aside. He is big on brain scanning (SPECT is his favorite protocol). He reported one research project where they were going to scan a large random sample of normal people. They preselected with a questionnaire to reject addicts, diagnosed mentally-ill, people with family history of Alzheimer’s and Parkinson’s, people with a memory of severe concussion, &c. Then they winnowed their population again with a preliminary brain scan. Of the first selected set, only ten percent of the people measured normal on their first brain scans and were kept in the study group!
This is an anecdotal data point for the contention that “normal” may be a dubious concept. This may be the single biggest flaw in medical research. People vary widely and they will necessarily vary widely in how they behave under experimental study. The normal distribution may be highly unusual for many practical applications.
If it won’t complicate your social life too much here, I’m quite curious about your ideas on diet—and if you don’t want to make them public to LW, could you PM me?
The “textbook stomach” compared to the range of actual stomachs.
I saw that chart a long time ago in an article in a vegetarian magazine. The article said that about 10% of the people who try vegetarianism don’t thrive on it. It also said that no one needs more than 4 oz. of meat per day to be healthy.l
A little more about variation—I’ve talked with a man who keeps records of his blood tests based on the hope that rejuvenation may be possible, and the theory that if it is, it will be very useful to know what blood chemistry details should be used as a target—some blood factors (sorry, I didn’t ask which ones) vary by a factor of ten among healthy people.
I find the paleo argument pretty compelling by virtue of the logic of the argument—i.e. homo sapiens evolved in an environment which did not have agriculture and grains. Also there is an M.D. blogger whose name escapes me who strongly advises people to avoid all processed grains because they pulse the glucose-insulin cycle in the blood. In March I overhauled my diet. I was eating a croissant every morning, two slices of bread every noon, and two to four cookies every evening. My grain consumption is now less than three slices of bread every four days. I eat cheese and yogurt. So I am not paleo. I do eat a serving of fruit and a serving of raw nuts at every meal.
I may be the most paleo person I know.
The other overhaul I did to my diet is the largest mass of food I am now taking in daily is a big plate of frozen vegetables at noon. I buy the two pound sacks of: 1) peas; 2) corn; 3) string beans; 4) broccoli and cauliflower mix; 5) peas and corn and carrots and string beans and lima beans mix. Choosing which of the sacks to pour from at noon is one of the high points in my typical day. My weight is down; my body mass index is down; my workouts have more pep. I am going to be hanging with this diet for a while.
I like Pollan’s formula: eat food, not too much, mostly plants. If somebody put that in the rationality quote thread I would upvote it. Beyond his formula, I do not like Pollan at all. He is not a nutritionist; he is a journalist. When I try and read any of his writing at length my eyes roll.
The best partisan of the paleo philosophy in my mind is the anthropologist David Abram who is part of the “re-wilding movement”. I am not anywhere close to a re-wilding movement partisan, but his discussions of forager cultures and what those folks may have to teach us makes Michael Pollan and most of the other more popular figures look like they are serving up extremely thin gruel.
Normal testosterone in males varies by a factor of almost 4:241-827 ng/dL. This sticks in my memory because the last time my doctor did my blood I was at the top of the normal range and I asked him how come I don’t look like a gorilla. (The 241-827 does not stick in my memory—I looked it up.) I think my doctor is awesome but his answer to that question was not.
Your textbook stomach link is truly a picture worth a thousand words.
The thing is, though, we have things like milk and alcohol that are not significantly older than grain consumption but we’ve seen evolutionary effects from them.
It is a point of scholarly contention whether people who retain their lactose tolerance with age were able to displace other people through their superior protein income, or taught milk cultivation to other people, who then had increased prevalence of the lactose toleration gene from that selection pressure. I have no more knowledge than the scholars, but consider the existence of a scholarly controversy evidence for the belief I am disposed against.
There’s also significant selection pressure against alcoholism genes among people that have access to alcohol (or just for ‘alcohol tolerance,’ but the first seems a more robust way to put things) and unsanitary drinking water: the result is that European Americans have dramatically lower rates of alcoholism than Native Americans. (The numbers I’ve seen compare alcoholism rates, not genes linked to alcoholism; so I suspect this is relevant but am not sure.)
So, it seems likely to me that people with European ancestry, at least, are likely to have spent the last hundred generations or so with a high percentage of grain in their diet, and we’ve seen that can make adjustments to patterns adopted for the previous thousand generations.
As for paleo? My friends that have tried it have all reported positive effects. My guess is that the main effects are better food and better food discipline. Whenever you make a serious attempt to plan your diet, some things disappear which you did not consider before- and my guess is that makes a huge difference. Extensive research shows that a lot of staples of industrial food- like sucrose, or massive levels of corn- are pretty bad for you. Simply preparing things yourself over having them prepared commercially has shown to dramatically reduce calorie intake; it’s easy to go to a restaurant and eat the food that tastes great when you didn’t see the two sticks of butter go into the pan. Beyond that, sticking to any sort of plan with food will decrease bad snacking and increase general good habits.
So, my advice is, “don’t pick a clearly deficient diet, and don’t not pick (i.e. default diet)”- beyond that, it doesn’t seem like you can do much besides match to your individual tastes. I think it would take a health benefit of 10 extra years for me to give up bread, since I really like bread. (And I’m expecting to live ~100 more years anyway, and so the duration effect has to be pretty massive to counterbalance the quality of life effect.)
That said, I suspect at some point in the next few years I’ll buy a blood glucose monitor and see what sort of effects my diet is having, and if I need to make any changes to prevent diabetes.
I hear you. Paleo = NO bread. I am not choosing to go that far. Seven months ago I had grains at the base of the food pyramid at ~ 40% of my daily calorie intake. Now it is less than 5% and the idea of eating a foot long subway is not appetizing to me at all right now. I am genetically European and I eat dairy and I consume alcohol; it is interesting and disturbing to observe at first hand people who dairy and alcohol make physically ill because they have some different G-A-T-C sequences deep in the works of the organism.
It is a mystery at this point. I have the latest American Dietetic Association Complete Food and Nutrition Guide. They have the food pyramid with grains at the base just like they showed us all in elementary school. My recent experience is that for my bio makeup that pyramid is not the best guide.
At least some people who are into paleo think that 80% or 90% will give you almost all the benefits.
Paleo diets generally consider corn a grain so you might want to avoid that. Some paleo variants (like the one I’m currently following) are ok with cheese and yogurt in moderation (and butter).
And there are false negatives as well as false positives.
See also: What’s In Your Placebo?
I’ve often worried about medical research because human trials are necessarily so small. And they’re expensive to run, so there seems to be an incentive only to conduct an experiment only when it will win acclaim for the experimenter.
I wonder if medicine needs to be supplemented with things that are not clinical trials—simulation runs or Brin-style big-but-sloppy survey-data studies. These are not a replacement for medical science, but they are at the very least a sanity check. And sanity checks are cheap and quick, so it’s easier to ask researchers to do a sanity check than to spend lots of time and money replicating another clinical trial.
While a small study is more likely to be wrong, there are two corollaries to that. The first is the second point raised—effect size. If (to take the extreme case) you have a study of three people who are all eighty years old and in a persistent vegetative state, and you give them Wundadrug, and they all wake up and suddenly look thirty years old, then you don’t need a much larger trial to see that Wundadrug is almost certainly doing something interesting.
More importantly in this context, they’re easier to replicate. If a study involves a hundred thousand patients over a thirty year period, then realistically nobody is ever going to check to see if they can get the same results. If, on the other hand, it involves ten patients for two weeks, anyone who is interested enough can try to replicate it with minimal effort.
Related: Reliability of ‘new drug target’ claims called into question:
Related reading: http://www.gwern.net/DNB%20FAQ#flaws-in-mainstream-science-and-psychology
Why do you write “Flaws in mainstream science”, if you mean specific parts of science only?
Some other mainstream areas have replication rates of more than 95%.
A specific part of science is part of mainstream science—or is a white horse not a horse?
If something applies to white horses only, I would write “white horses” instead of “horses”. Otherwise it might suggest (at least to some readers) that it applies to many, most or even all horses. It is not wrong, but it can be misleading.
I’m not sure it is misleading; the material is obviously focused on health and psychology as the areas I read most in, but the the results I discuss should apply to many areas: the specific problems of no incentives for replication or less than p<0.05 significance are common to all areas or all areas which use NHST statistics, etc. You may like to think that hard sciences like chemistry are exempt… but I get a lot of these citations off a biochemistry blog!
Papers I read are mainly physics papers, especially particle physics. Not replicated results there are so rare that they often get significant attention in the community (Blog article) or even mainstream media (OPERA neutrino speed measurement).
The usual study&publication process for a new particle detector looks like that:
identify particles flying through the detector (known for >50 years)
find the decays of frequent short-living particles (known for >30 years), use them as calibration
look for other known particles and compare their masses and decays with the existing values
look for known decay modes of those particles and related properties, compare them with existing values and improve them by a significant factor
find new things
Completely new measurements are just a small fraction of the studies—most results confirm earlier experiments and improve the precision.
Why Most Published Research Findings Are False, 2005
Potential pitfalls in the use of p-values and in interpretation of significance levels: http://dx.doi.org/10.1016/0167-8140(94)90072-8, 1994
Hans-Peter Beck-Bornholdt and Hans-Hermann Dubben are writing books and papers on this problem (most medical research is wrong, see “Is the pope an alien?” Nature 381: 730, 1996) for decades. I’ve even got a popular science book on probability by them. Sadly most stuff is only available in German.
Curiously, Beck-Bornholdt and Dubben seem (at least, in the two papers of theirs you cited) unaware of the Bayesian solution to the problem they pose. In fact, in “Potential pitfalls...”, the abstract concludes with “more care in the use of p-values in analysis and interpretation of clinical data is required”, which is a bit like saying that the patient bled to death, so more care in the use of blood-letting is required.
That may just reflect the mid-1990s, though. “Is the Pope an alien?” drew these replies in Nature (vol.382 p.480), only one of which gives the Bayesian solution, and adds that “It is a shame that Bayesian methods are not part of all introductory statistics classes.” I don’t know if they are now.
(BTW, you need to backslash the parentheses in your DOI URL to make it work with markdown.)
I’m a grad student/TA in the statistics department at Iowa State University. Bayes doesn’t make it into any of our intro classes outside of maybe introducing Bayes’ theorem… but I doubt it. It also doesn’t make it into our stat classes for non-major grad students. Some non-stat major grad students take master’s level stat courses instead. They introduce some Bayesian material depending on the prof, but not much. There is also a master’s level course in Bayesian methods, and new PhD level Bayesian courses in both methods and theory.
Relevant info: our dept. is in the top 10 in the US and so relatively typical, but it is also known for being a bastion of frequentism. Bayesians are making inroads in the department though, but intro classes at Duke, for example, might be much more Bayesian.
Working link: http://dx.doi.org/10.1016/0167-8140\(94\)90072-8
Someone even wrote a whole book that demonstrates the misuse and failure in the use of probability: The Astrology File: Scientific Proof of the Link Between Star Signs and Human Behavior
Hmmm. The customer reviews seem to suggest that it wasn’t interpreted as “a mock”.
I was particularly amused by the list of what customers eventually purchased after looking at the amazon page for this book:
Yes, I don’t know if it is one (I just hope it is :-). But in one of the books by Beck-Bornholdt and Dubben it is interpreted as a mock or at least a good example of how you can prove everything you like if you misuse statistics or your knowledge of probability is sufficiently crippled.
I didn’t see any evidence on the page, either in the editorial or customer reviews, that this was intended as a “mock”. It looks like a seriously intended book of pseudo-science from the Amazon page.
http://www.inference.phy.cam.ac.uk/mackay/pope.html
Do dead salmon dream of electric sheep? (Related article)
Incidentally, something like the paleo diet works incredibly well for me. I discovered the right details for me about 4 months ago and have quickly lost weight to a point near my optimal weight and I don’t have dizzy or tired spells anymore with blood sugar fluctuations. However, calling it the ‘paleo’ diet begs a question I don’t have enough background in evolutionary biology to ask, but I can pose a related question, because it is more concrete:
My father-in-law is on some kind of fruit diet, and he raves about how healthy fruits are, especially if they are fresh and eaten with the peel. Apples, for example, have all these different healthy things in them.
I wonder why an apple should be healthy. Wouldn’t any animal be satisfied with a fruit that just had calories? Enough, in any case, to come back for more and scatter the seeds? Why should an apple—or anything ‘natural’—be so especially healthy for humans?
You’re missing the other side of the story. Humans evolved to obtain their nutritional needs from those foods that were available in the EEA and this effect is probably more significant than the selection pressure in the other direction (on fruits to be nutritionally beneficial to animals that eat them). Humans are adapted to a diet that includes things that were available to them during the long pre-agricultural evolutionary period.
OK. So reading between the lines somewhat and pushing the argument further, would it similarly/analogously/generally be the case that eating nearly any part of an evolutionarily ‘old’, highly evolved autotroph would usually be healthy, because they have learned how to make things that are useful for living things to have? (That is, ignoring the part of the argument possibly implying that we might have evolved a dependency upon apples in particular, because that seemed unlikely to me. Though I could be convinced because know it is the case to some degree for oranges and scurvy, for example.)
I don’t think that is correct. Much autotroph biomass is indigestible (think tree trunks) and some of the rest is deliberately poisonous. A clever plant doesn’t want to be eaten. Furthermore, we ought to eat stuff that has good ingredients—we don’t really care whether it makes them itself, or steals them from the manufacturer. And in any case, different living things are built from slightly different kinds of stuff—a lot of autotrophs don’t even bother with vitamins A, C, D, and E.
And finally, try to avoid using the terms “evolutionarily old” and “highly evolved”. Every living thing you see around you is equally “evolved”, since every living thing has the same “evolutionary age”—roughly 3.5 billion years.
Surely number of generations matters, too?
I’m not sure that this is at all meaningful because things like selection pressure and mutation rate also arguably matter. If one has a species with lots of generations but an incredibly low mutation rate it isn’t going to adapt to an environment as much as another species in the same environment with a higher mutation rate.
Ok, then. Number of generations. Which makes human among the least evolved of all species on the planet. Well, maybe some tortoises are less evolved than us, and maybe elephants and whales and sequoias too, but we certainly have evolved through fewer generations than rats, jellyfish, mosquitoes, sunflowers, earthworms, amoebae, and E. coli.
OK, corrected, thanks. I definitely see the problem with ‘highly evolved’. (For example, a “highly evolved” organism could lose and gain the ability to make a vitamin many times over). I was having trouble separating the ideas of ‘older’ (as in chronologically first) and ‘autotroph’ (as in independent). Animals don’t dependably make the vitamins they need from plants because they can get these vitamins from plants. Plants, though, couldn’t and can’t depend upon something else producing them, so they make them on their own.
Yes, of course. It was sloppy of me not to add the qualifier, ‘if edible’. Instead, what about the validity of this statement: regardless of its edibility, and the amount and type of toxins that might be present, any plant part would be expected to have as many vitamins as a fruit part? The argument being that plants did not have evolutionary pressure to make their fruits particularly full of vitamins?
Agreed. My question is why the former would be healthier. Perhaps because manufacturers sequester more, in greater concentrations?
Animals have abilities to detect whether what they eat is nutritious, plants give the animals what they want. That includes things like Vitamin C—though that isn’t an essential nutrient for most animals.
E.g. see: http://en.wikipedia.org/wiki/Specific_appetite
As mattnewport mentioned above our ancestors evolved to live on fruits. Most animals can synthesis their own vitamin C. We only lost that ability because our ancestors had so much of it in their diet that they didn’t need to synthesize it.
If fruits didn’t contain vitamin C, we wouldn’t have lost the ability to synthesize it, possibly losing the ability to synthesize something else that they did have.
I don’t see any reason for thinking it would be true. Any plant part should be expected to have as many vitamins as it needs to do what that part of a plant does. Different vegetables are rich in different vitamins.
Fruits are relatively rich in vitamin C because they need antioxidants. Except for the seeds, they are relatively poor in B vitamins because they don’t conduct a lot of metabolic activity.
Who says that it is healthier? Oh, there are certainly arguments against dining too high on the food chain, but I am very uncomfortable with any blanket claim that vegetarianism is healthier than being omnivorous.
You mean the fruits themselves need antioxidants? That’s interesting! And would explain why fruits are high in antioxidants. What do they need them for?
To keep their (concentrated, moist) sugars from oxidizing, and effectively becoming caramel-like or tar-like, and hence less appetizing to the animals that are going to be tricked into distributing the seeds. Or rather, appetizing for too short a time.
Multinationals put anti-oxidants into junk food to promote shelf-life. Nature puts anti-oxidants into fruit to promote branch-life.
Warning! I might just be making this up. Check more authoritative sources if this info matters to you.
Edit: It occurs to me that the essential oils (fragrances) are also subject to being oxidized into something less savory than the original. Maybe more at risk of oxidizing than the sugars.
I would guess this is not true in general since many things do not want to be eaten and so evolve various defense mechanisms. In turn the organisms that eat them may develop counter-measures that enable them to safely digest their meal despite the defense mechanisms but this will depend on the complex evolutionary history of both organisms. Ruminants are adapted to a quite different diet than humans for example.
More on self-experimentation: http://www.quantifiedself.com/
And they’re wrong. Modern diet is a lot more paleo than diet before metabolic syndrome epidemic.
Strong correlation pointing the wrong way is a pretty good evidence for lack of causation.
Let me check this at the site. Quoting:
...
That’s hard to read because the numbers are all squashed together in a line, so I will rewrite based on the description given:
In 1961, the United States diet was 20.36% Paleolithic, 50.72% Neolithic, and 28.92% Industrial.
In 2007, the United States diet was 20.51% Paleolithic, 43.43% Neolithic, and 36.06% Industrial.
The United States diet is probably what pertains to most readers of this forum, and it certainly is what is usually being talked about in the American media when the American media goes on about the obesity epidemic.
But the modern United States diet is not “a lot more Paleo”. That’s a wildly wrong summary of what the data shows. Rather, the modern diet is a lot more Industrial. Specifically, the modern diet has a lot more sweeteners and vegetable oils.
The summary “they’re wrong...modern diet is a lot more paleo...” creates the impression that we have largely taken paleo advice and have suffered greatly for it. But in the case of the United States, where the obesity epidemic is massive, that is simply not the case. The big change is in consuming more sweeteners and more vegetable oils—according to the data presented.
Moreover, my impression—confirmed when googling this—is that the warning against sweeteners and vegetable oils is an important part of the paleo critique of the modern diet. If you follow paleo advice, you will abandon sweeteners and vegetable oils.
So, the correlation does not “go the wrong way”, in the case of the United States. There are a lot of other countries listed there, but since I don’t know whether and how much they are subject to an obesity epidemic, I can’t make use of the data.
You’re wrong. 1961 USA was already quite far in its transition away from Neolithic food, it just takes time for such low-level damage to accumulate.
There’s little high quality world-wide data before 1961, but feel free to offer a bet about it if you doubt what it would reach.
Correlation goes the wrong way, just like I said.
In any case, obesity statistics are easily available for all countries, so I’m puzzled by your unwillingness to take a look.
It’s not. Paleos bundle all non-Paleo foods as if they were pretty much the same. It’s about as useful as abstinence-only approach to safe sex.
No, paleo dieters don’t bundle all non-paleo foods. Almost all foods that can be bought in a store or farmers market has been altered in one way or another since the neolithic began. And most paleo dieters acknowledge this. The purpose is not to perfectly replicate the diet of a paleolithic human, in the fashion of a civil war reenactor, but to most closely mimic the profile of the diet humans evolved to eat.
Some factors in this are nutrient density, lower simple carbs, intermittent fasting, and ketosis. These have been shown in research to be beneficial. But of course, there is a lot of research that shows a diversity of results. The strongest evidence comes from studies of hunter-gatherers who mostly lack chronic diseases, and this forms a central basis of the paleo diet.
By the way, Americans ate more meat than bread prior to the 20th century. They also only ate a fraction of the sugar in the past. Yet many of the chronic diseases only became common in the 20th century. Heart disease was a rare disease in the early 20th century, but by the end of the century it was rampant.