[EDIT 10/27: Slight changes in response to feedback; correcting some definitions. I am not an expert in this field and will continue to make changes as I learn about them. There is a critique of this post here and other worse critiques elsewhere. My only excuse for doing this is that I am failing less spectacularly than other online sources writing about the same topic.]
I’ve worked with doctors who think Alcoholics Anonymous is so important for the treatment of alcoholism that anyone who refuses to go at least three times a week is in denial about their problem and can’t benefit from further treatment.
I’ve also worked with doctors who are so against the organization that they describe it as a “cult” and say that a physician who recommends it is no better than one who recommends crystal healing or dianetics.
I finally got so exasperated that I put on my Research Cap and started looking through the evidence base.
My conclusion, after several hours of study, is that now I understand why most people don’t do this.
The studies surrounding Alcoholics Anonymous are some of the most convoluted, hilariously screwed-up research I have ever seen. They go wrong in ways I didn’t even realize research could go wrong before. Just to give some examples:
– In several studies, subjects in the “not attending Alcoholics Anonymous” condition attended Alcoholics Anonymous more than subjects in the “attending Alcoholics Anonymous” condition.
– Almost everyone’s belief about AA’s retention rate is off by a factor of five because one person long ago misread a really confusing graph and everyone else copied them without double-checking.
– The largest study ever in the field, a $30 million effort over 8 years following thousands of patients, had no untreated control group.
Not only are the studies poor, but the people interpreting them are heavily politicized. The entire field of addiction medicine has gotten stuck in the middle of some of the most divisive issues in our culture, like whether addiction is a biological disease or a failure of willpower, whether problems should be solved by community and peer groups or by highly trained professionals, and whether there’s a role for appealing to a higher power in any public organization. AA’s supporters see it as a scruffy grassroots organization of real people willing to get their hands dirty, who can cure addicts failed time and time again by a system of glitzy rehabs run by arrogant doctors who think their medical degrees make them better than people who have personally fought their own battles. Opponents see it as this awful cult that doesn’t provide any real treatment and just tells addicts that they’re terrible people who will never get better unless they sacrifice their identity to the collective.
As a result, the few sparks of light the research kindles are ignored, taken out of context, or misinterpreted.
The entire situation is complicated by a bigger question. We will soon find that AA usually does not work better or worse than various other substance abuse interventions. That leaves the sort of question that all those fancy-shmancy people with control groups in their studies don’t have to worry about – does anything work at all?
I.
We can start by just taking a big survey of people in Alcoholics Anonymous and seeing how they’re doing. On the one hand, we don’t have a control group. On the other hand…well, there really is no other hand, but people keep doing it.
According to AA’s own surveys, one-third of new members drop out by the end of their first month, half by the end of their third month, and three-quarters by the end of their first year. “Drop out” means they don’t go to AA meetings anymore, which could be for any reason including (if we’re feeling optimistic) them being so completely cured they no longer feel they need it.
There is an alternate reference going around that only 5% (rather than 25%) of AA members remain after their first year. This is a mistake caused by misinterpreting a graph showing that only five percent of members in their first year were in their twelfth month of membership, which is obviously completely different. Nevertheless, a large number of AA hate sites (and large rehabs!) cite the incorrect interpretation, for example the Orange Papers and RationalWiki’s page on Alcoholics Anonymous. In fact, just to keep things short, assume RationalWiki’s AA page makes every single mistake I warn against in the rest of this article, then use that to judge them in general. On the other hand, Wikipedia gets it right and I continue to encourage everyone to use it as one of the most reliable sources of medical information available to the public (I wish I was joking).
This retention information isn’t very helpful, since people can remain in AA without successfully quitting drinking, and people may successfully quit drinking without being in AA. However, various different sources suggest that, of people who stay in AA a reasonable amount of time, about half stop being alcoholic. These numbers can change wildly depending on how you define “reasonable amount of time” and “stop being alcoholic”. Here is a table, which I have cited on this blog before and will probably cite again:
Behold. Treatments that look very impressive (80% improved after six months!) turn out to be the same or worse as the control group. And comparing control group to control group, you can find that “no treatment” can appear to give wildly different outcomes (from 20% to 80% “recovery”) depending on what population you’re looking at and how you define “recovery”.
Twenty years ago, it was extremely edgy and taboo for a reputable scientist to claim that alcoholics could recover on their own. This has given way to the current status quo, in which pretty much everyone in the field writes journal articles all the time about how alcoholics can recover on their own, but make sure to harp upon how edgy and taboo they are for doing so. From these sorts of articles, we learn that about 80% of recovered alcoholics have gotten better without treatment, and many of them are currently able to drink moderately without immediately relapsing (something else it used to be extremely taboo to mention). Kate recently shared an good article about this: Most People With Addiction Simply Grow Out Of It: Why Is This Widely Denied?
Anyway, all this stuff about not being able to compare different populations, and the possibility of spontaneous recovery, just mean that we need controlled experiments. The largest number of these take a group of alcoholics, follow them closely, and then evaluate all of them – the AA-attending and the non-AA-attending – according to the same criteria. For example Morgenstern et al (1997), Humphreys et al (1997) andMoos (2006). Emrick et al (1993) is a meta-analyses of a hundred seventy three of these. All of these find that the alcoholics who end up going to AA meetings are much more likely to get better than those who don’t. So that’s good evidence the group is effective, right?
Bzzzt! No! Wrong! Selection bias!
People who want to quit drinking are more likely to go to AA than people who don’t want to quit drinking. People who want to quit drinking are more likely to actually quit drinking than those who don’t want to. This is a serious problem. Imagine if it is common wisdom that AA is the best, maybe the only, way to quit drinking. Then 100% of people who really want to quit would attend compared to 0% of people who didn’t want to quit. And suppose everyone who wants to quit succeeds, because secretly, quitting alcohol is really easy. Then 100% of AA members would quit, compared to 0% of non-members – the most striking result it is mathematically possible to have. And yet AA would not have made a smidgeon of difference.
But it’s worse than this, because attending AA isn’t just about wanting to quit. It’s also about having the resources to make it to AA. That is, wealthier people are more likely to hear about AA (better information networks, more likely to go to doctor or counselor who can recommend) and more likely to be able to attend AA (better access to transportation, more flexible job schedules). But wealthier people are also known to be better at quitting alcohol than poor people – either because the same positive personal qualities that helped them achieve success elsewhere help them in this battle as well, or just because they have fewer other stressors going on in their lives driving them to drink.
Finally, perseverance is a confounder. To go to AA, and to keep going for months and months, means you’ve got the willpower to drag yourself off the couch to do a potentially unpleasant thing. That’s probably the same willpower that helps you stay away from the bar.
And then there’s a confounder going the opposite direction. The worse your alcoholism is, the more likely you are to, as the organization itself puts it, “admit you have a problem”.
These sorts of longitudinal studies are almost useless and the field has mostly moved away from them. Nevertheless, if you look on the pro-AA sites, you will find them in droves, and all of them “prove” the organization’s effectiveness.
III.
It looks like we need randomized controlled trials. And we have them. Sort of.
is the study beloved of the AA hate groups, since it purports to show that people in Alcoholics Anonymous not only don’t get better, but are nine times more likely to binge drink than people who don’t go into AA at all.
There are a number of problems with this conclusion. First of all, if you actually look at the study, this is one of about fifty different findings. The other findings are things like “88% of treated subjects reported a reduction in drinking, compared to 50% of the untreated control group”.
Second of all, the increased binge drinking was significant at the 6 month followup period. It was not significant at the end of treatment, the 3 month followup period, the 9 month followup period, or the 12 month followup period. Remember, taking a single followup result out of the context of the other followup results is a classic piece of Dark Side Statistics and will send you to Science Hell.
Of multiple different endpoints, Alcoholics Anonymous did better than no treatment on almost all of them. It did worse than other treatments on some of them (dropout rates, binge drinking, MMPI scale) and the same as other treatments on others (abstinent days, total abstinence).
If you are pro-AA, you can say “Brandsma study proves AA works!”. If you are anti-AA, you can say “Brandsma study proves AA works worse than other treatments!”, although in practice most of these people prefer to quote extremely selective endpoints out of context.
However, most of the patients in the Brandsma study were people convicted of alcohol-related crimes ordered to attend treatment as part of their sentence. Advocates of AA make a good point that this population might be a bad fit for AA. They may not feel any personal motivation to treatment, which might be okay if you’re going to listen to a psychologist do therapy with you, but fatal for a self-help group. Since the whole point of AA is being in a community of like-minded individuals, if you don’t actually feel any personal connection to the project of quitting alcohol, it will just make you feel uncomfortable and out of place.
Also, uh, this just in, Brandsma didn’t use a real AA group, because the real AA groups make people be anonymous which makes it inconvenient to research stuff. He just sort of started his own non-anonymous group, let’s call it A, with no help from the rest of the fellowship, and had it do Alcoholics Anonymous-like stuff. On the other hand, many members of his control group went out into the community and…attended a real Alcoholics Anonymous, because Brandsma can’t exactly ethically tell them not to. So technically, there were more people in AA in the no-AA group than in the AA group. Without knowing more about Alcoholics Anonymous, I can’t know whether this objection is valid and whether Brandsma’s group did or didn’t capture the essence of the organization. Still, not the sort of thing you want to hear about a study.
Walsh et al (1991) is a similar study with similar confounders and similar results. Workers in an industrial plant who were in trouble for coming in drunk were randomly assigned either to an inpatient treatment program or to Alcoholics Anonymous. After a year of followup, 60% of the inpatient-treated workers had stayed sober, but only 30% of the AA-treated workers had.
The pro-AA side made three objections to this study, of which one is bad and two are good.
The bad objection was that AA is cheaper than hospitalization, so even if hospitalization is good, AA might be more efficient – after all, we can’t afford to hospitalize everyone. It’s a bad objection because the authors of the study did the math and found out that hospitalization was so much better than AA that it decreased the level of further medical treatment needed and saved the health system more money than it cost.
The first good objection: like the Brandsma study, this study uses people under coercion – in this case, workers who would lose their job if they refused. Fine.
The second good objection, and this one is really interesting: a lot of inpatient hospital rehab is AA. That is, when you go to an hospital for inpatient drug treatment, you attend AA groups every day, and when you leave, they make you keep going to the AA groups. In fact, the study says that “at the 12 month and 24 month assessments, the rates of AA affiliation and attendance in the past 6 months did not differ significantly among the groups.” Given that the hospital patients got hospital AA + regular AA, they were actually getting more AA than the AA group!
So all that this study proves is that AA + more AA + other things is better than AA. There was no “no AA” group, which makes it impossible to discuss how well AA does or doesn’t work. Frick.
Timko (2006) is the only study I can hesitantly half-endorse. This one has a sort of clever methodological trick to get around the limitation that doctors can’t ethically refuse to refer alcoholics to treatment. In this study, researchers at a Veterans’ Affairs hospital randomly assigned alcoholic patients to “referral” or “intensive referral”. In “referral”, the staff asked the patients to go to AA. In “intensive referral”, the researchers asked REALLY NICELY for the patients to go to AA, and gave them nice glossy brochures on how great AA was, and wouldn’t shut up about it, and arranged for them to meet people at their first AA meeting so they could have friends in AA, et cetera, et cetera. The hope was that more people in the “intensive referral” group would end out in AA, and that indeed happened scratch that, I just re-read the study and the same number of people in both groups went to AA and the intensive group actually completed a lower number of the 12 Steps on average, have I mentioned I hate all research and this entire field is terrible? But the intensive referral people were more likely to have “had a spiritual awakening” and “have a sponsor”, so it was decided the study wasn’t a complete loss and when it was found the intensive referral condition had slightly less alcohol use the authors decided to declare victory.
So, whereas before we found that AA + More AA was better than AA, and that proved AA didn’t work, in this study we find that AA + More AA was better than AA, and that proves AA does work. You know, did I say I hesitantly half-endorsed this study? Scratch that. I hate this study too.
IV.
All right, @#%^ this $@!&*. We need a real study, everything all lined up in a row, none of this garbage. Let’s just hire half the substance abuse scientists in the country, throw a gigantic wad of money at them, give them as many patients as they need, let them take as long as they want, but barricade the doors of their office and not let them out until they’ve proven something important beyond a shadow of a doubt.
This was about how the scientific community felt in 1989, when they launched Project MATCH. This eight-year, $30 million dollar, multi-thousand patient trial was supposed to solve everything.
The people going into Project MATCH might have been a little overconfident. Maybe “not even Zeus could prevent this study from determining the optimal treatment for alcohol addiction” overconfident. This might have been a mistake.
The study was designed with three arms, one for each of the popular alcoholism treatments of the day. The first arm would be “twelve step facilitation”, a form of therapy based off of Alcoholics Anonymous. The second arm would be cognitive behavioral therapy, the most bog-standard psychotherapy in the world and one which by ancient tradition must be included in any kind of study like this. The third arm would be motivational enhancement therapy, which is a very short intervention where your doctor tells you all the reasons you should quit alcohol and tries to get you to convince yourself.
There wasn’t a “no treatment” arm. This is where the overconfidence might have come in. Everyone knew alcohol treatment worked. Surely you couldn’t dispute that. They just wanted to see which treatment worked best for which people. So you would enroll a bunch of different people – rich, poor, black, white, married, single, chronic alcoholic, new alcoholic, highly motivated, unmotivated – and see which of these people did best in which therapy. The result would be an algorithm for deciding where to send each of your patients. Rich black single chronic unmotivated alcoholic? We’ve found with p < 0.00001 that the best place for someone like that is in motivational enhancement therapy. Such was the dream. So, eight years and thirty million dollars and the careers of several prestigious researchers later, the results come in, and—yeah, everyone does exactly the same on every kind of therapy (with one minor, possibly coincidental exception). Awkward. “Everybody has won and all must have prizes!”. If you’re an optimist, you can say all treatments work and everyone can keep doing whatever they like best. If you’re a pessimist, you might start wondering whether anything works at all.
By my understanding this is also the confusing conclusion of Ferri, Amato & Davoli (2006), the Cochrane Collaboration’s attempt to get in on the AA action. Like all Cochrane Collaboration studies since the beginning of time, they find there is insufficient evidence to demonstrate the effectiveness of the intervention being investigated. This has been oft-quoted in the anti-AA literature. But by my reading, they had no control groups and were comparing AA to different types of treatment:
Three studies compared AA combined with other interventions against other treatments and found few differences in the amount of drinks and percentage of drinking days. Severity of addiction and drinking consequence did not seem to be differentially influenced by TSF versus comparison treatment interventions, and no conclusive differences in treatment drop out rates were reported.
So the two best sources we have – Project MATCH and Cochrane – don’t find any significant differences between AA and other types of therapy. Now, to be fair, the inpatient treatment mentioned in Walsh et al wasn’t included, and inpatient treatment might be the gold standard here. But sticking to various forms of outpatient intervention, they all seem to be about the same.
So, the $64,000 question: do all of them work well, or do all of them work poorly?
V.
Alcoholism studies avoid control groups like they are on fire, presumably because it’s unethical not to give alcoholics treatment or something. However, there is one class of studies that doesn’t have that problem. These are the ones on “brief opportunistic intervention”, which is much like a turbocharged even shorter version of “motivational enhancement therapy”. Your doctor tells you ‘HELLO HAVE YOU CONSIDERED QUITTING ALCOHOL??!!’ and sees what happens.
Brief opportunistic intervention is the most trollish medical intervention ever, because here are all these brilliant psychologists and counselors trying to unravel the deepest mysteries of the human psyche in order to convince people to stop drinking, and then someone comes along and asks “Hey, have you tried just asking them politely?”. And it works.
Not consistently. But it works for about one in eight people. And the theory is that since it only takes a minute or two of a doctor’s time, it scales a lot faster than some sort of hideously complex hospital-based program that takes thousands of dollars and dozens of hours from everyone involved. If doctors would just spend five minutes with each alcoholic patient reminding them that no, really, alcoholism is really bad, we could cut the alcoholism rate by 1⁄8.
(this also works for smoking, by the way. I do this with every single one of my outpatients who smoke, and most of the time they roll their eyes, because their doctor is giving them that speech, but every so often one of them tells me that yeah, I’m right, they know they really should quit smoking and they’ll give it another try. I have never saved anyone’s life by dramatically removing their appendix at the last possible moment, but I have gotten enough patients to promise me they’ll try quitting smoking that I think I’ve saved at least one life just by obsessively doing brief interventions every chance I get. This is probably the most effective life-saving thing you can do as a doctor, enough so that if you understand it you may be licensed to ignore 80,000 Hours’ arguments on doctor replaceability)
Anyway, for some reason, it’s okay to do these studies with control groups. And they are so fast and easy to study that everyone studies them all the time. A meta-analysis of 19 studies is unequivocal that they definitely work.
Why do these work? My guess is that they do two things. First, they hit people who honestly didn’t realize they had a problem, and inform them that they do. Second, the doctor usually says they’ll “follow up on how they’re doing” the next appointment. This means that a respected authority figure is suddenly monitoring their drinking and will glare at them if they stay they’re still alcoholic. As someone who has gone into a panic because he has a dentist’s appointment in a week and he hasn’t been flossing enough – and then flossed until his teeth were bloody so the dentist wouldn’t be disappointed – I can sympathize with this.
But for our purposes, the brief opportunistic intervention sets a lower bound. It says “Here’s a really minimal thing that seems to work. Do other things work better than this?”
The “brief treatment” is the next step up from brief intervention. It’s an hour-or-so-long session (or sometimes a couple such sessions) with a doctor or counselor where they tell you some tips for staying off alcohol. I bring it up here because the brief treatment research community spends its time doing studies that show that brief treatments are just as good as much more intense treatments. This might be most comparable to the “motivational enhancement therapy” in the MATCH study.
Chapman and Huygens (1988) find that a single interview with a health professional is just as good as six weeks of inpatient treatment (I don’t know about their hospital in New Zealand, but for reference six weeks of inpatient treatment in my hospital costs about $40,000.)
Edwards (1977) finds that in a trial comparing “conventional inpatient or outpatient treatment complete with the full panoply of services available at a leading psychiatric institution and lasting several months” versus an hour with a doc, both groups do the same at one and two year followup.
. We have all of these different categories of things, and some of them are the same as others and others are bigger than others but we’re not sure exactly where all of them stand.
We have classes “no treatment”, “brief opportunistic intervention”, “brief treatment”, “Alcoholics Anonymous”, “psychotherapy”, and “inpatient”.
We can prove that BOI > NT, and that AA = PT. Also that BT = IP = PT. We also have that IP > AA, which unfortunately we can use to prove a contradiction, so let’s throw it out for now.
So the hierarchy of classes seems to be (NT) < (BOI) ? (BT, IP, AA, PT) - in other words, no treatment is the worst, brief opportunistic intervention is better, and then somewhere in there we have this class of everything else that is the same.
Can we prove that BOI = BT?
We have some good evidence for this, once again from our Handbook. A study in Edinburgh finds that five minutes of psychiatrist advice (brief opportunistic intervention) does the same as sixty minutes of advice plus motivational interviewing (brief treatment).
So if we take all this seriously, then it looks like every psychosocial treatment (including brief opportunistic intervention) is the same, and all are better than no treatment. This is a common finding in psychiatry and psychology – for example, all common antidepressants are better than no treatment but work about equally well; all psychotherapies are better than no treatment but work about equally well, et cetera. It’s still an open question what this says about our science and our medicine.
The strongest counterexample to this is Walsh et al which finds the inpatient hospital stay works better than the AA referral, but this study looks kind of lonely compared to the evidence on the other side. And even the authors admit they were surprised by the effectiveness of the hospital there.
And let’s go back to Project MATCH. There wasn’t a control group. But there were the people who dropped out of the study, who said they’d go to AA or psychotherapy but never got around to it. Cutter and Fishbain (2005) take a look at what happened to these folks. They find that the dropouts did 75% as well as the people in any of the therapy groups, and that most of the effect of the therapy groups occurred in the first week (ie people dropped out after one week did about 95% as well as people who stayed in).
To me this suggests two things. First, therapy is only a little helpful over most people quitting on their own. Second, insofar as therapy is helpful, the tiniest brush with therapy is enough to make someone think “Okay, I’ve had some therapy, I’ll be better now”. Just like with the brief opportunistic interventions, five minutes of almost anything is enough.
This is a weird conclusion, but I think it’s the one supported by the data.
VI.
I should include a brief word about this giant table.
, the author of which is known as a very careful scholar whom I cannot help but respect.
And the table does a good thing in discussing medications like acamprosate and naltrexone, which are very important and effective interventions but which will not otherwise be showing up in this post.
However, the therapy part of the table looks really wrong to me.
First of all, I notice acupuncture is ranked 17 out of 48, putting in a much, much better showing than treatments like psychotherapy, counseling, or education. Seems fishy.
Second of all, I notice that motivational enhancement (#2), cognitive therapy (#13), and twelve-step (#37) are all about as far apart as could be, but the largest and most powerful trial ever, Project MATCH, found all three to be about equal in effectiveness.
Third of all, I notice that cognitive therapy is at #13, but psychotherapy is at #46. But cognitive therapy is a kind of psychotherapy.
Fourth of all, I notice that brief interventions, motivational enhancement, confrontational counseling, psychotherapy, general alcoholism counseling, and education are all over. But a lot of these are hard to differentiate from one another.
The table seems messed up to me. Part of it is because it is about evidence base rather than effectiveness (consider that handguns have a stronger evidence base than the atomic bomb, since they have been used many more times in much better controlled conditions, but the atomic bomb is more effective) and therefore acupuncture, which is poorly studied, can rank quite high compared to things which have even one negative study.
But part of it just seems wrong. I haven’t read the full book, but I blame the tendency to conflate studies showing “X does not work better than anything else” with “X does not work”.
Remember, whenever there are meta-analyses that contradict single very large well-run studies, go with the single very large well-run study, especially when the meta-analysis is as weird as this one. Project MATCH is the single very large well-run study, and it says this is balderdash. I’m guessing it’s trying to use some weird algorithmic methodology to automatically rate and judge each study, but that’s no substitute for careful human review.
VII.
In conclusion, as best I can tell – and it is not very well, because the studies that could really prove anything robustly haven’t been done – most alcoholics get better on their own. All treatments for alcoholism, including Alcoholics Anonymous, psychotherapy, and just a few minutes with a doctor explaining why she thinks you need to quit, increase this already-high chance of recovery a small but nonzero amount. Furthermore, they are equally effective after only a tiny dose: your first couple of meetings, your first therapy session. Some studies suggest that inpatient treatment with outpatient followup may be better than outpatient treatment alone, but other studies contradict this and I am not confident in the assumption.
So does Alcoholics Anonymous work? Though I cannot say anything authoritatively, my impression is: Yes, but only a tiny bit, and for many people five minutes with a doctor may work just as well as years completing the twelve steps. As such, individual alcoholics may want to consider attending if they don’t have easier options; doctors might be better off just talking to their patients themselves.
If this is true – and right now I don’t have much confidence that it is, it’s just a direction that weak and contradictory data are pointing – it would be really awkward for the multibazillion-dollar treatment industry.
More worrying, I am afraid of what it would do to the War On Drugs. Right now one of the rallying cries for the anti-Drug-War movement is “treatment, not prison”. And although I haven’t looked seriously at the data for any drug besides alcohol. I think some data there are similar. There’s very good medication for drugs – for example methadone and suboxone for opiate abuse – but in terms of psychotherapy it’s mostly the same stuff you get for alcohol. Rehabs, whether they work or not, seem to serve an important sort of ritual function, where if you can send a drug abuser to a rehab you at least feel like something has been done. Deny people that ritual, and it might make prison the only politically acceptable option.
In terms of things to actually treat alcoholism, I remain enamoured of the Sinclair Method, which has done crazy outrageous stuff like conduct an experiment with an actual control group. But I haven’t investigated enough to know whether my early excitement about them looks likely to pan out or not.
I would not recommend quitting any form of alcohol treatment that works for you, or refusing to try a form of treatment your doctor recommends, based on any of this information.
Alcoholics Anonymous: Much More Than You Wanted To Know
[EDIT 10/27: Slight changes in response to feedback; correcting some definitions. I am not an expert in this field and will continue to make changes as I learn about them. There is a critique of this post here and other worse critiques elsewhere. My only excuse for doing this is that I am failing less spectacularly than other online sources writing about the same topic.]
I’ve worked with doctors who think Alcoholics Anonymous is so important for the treatment of alcoholism that anyone who refuses to go at least three times a week is in denial about their problem and can’t benefit from further treatment.
I’ve also worked with doctors who are so against the organization that they describe it as a “cult” and say that a physician who recommends it is no better than one who recommends crystal healing or dianetics.
I finally got so exasperated that I put on my Research Cap and started looking through the evidence base.
My conclusion, after several hours of study, is that now I understand why most people don’t do this.
The studies surrounding Alcoholics Anonymous are some of the most convoluted, hilariously screwed-up research I have ever seen. They go wrong in ways I didn’t even realize research could go wrong before. Just to give some examples:
– In several studies, subjects in the “not attending Alcoholics Anonymous” condition attended Alcoholics Anonymous more than subjects in the “attending Alcoholics Anonymous” condition.
– Almost everyone’s belief about AA’s retention rate is off by a factor of five because one person long ago misread a really confusing graph and everyone else copied them without double-checking.
– The largest study ever in the field, a $30 million effort over 8 years following thousands of patients, had no untreated control group.
Not only are the studies poor, but the people interpreting them are heavily politicized. The entire field of addiction medicine has gotten stuck in the middle of some of the most divisive issues in our culture, like whether addiction is a biological disease or a failure of willpower, whether problems should be solved by community and peer groups or by highly trained professionals, and whether there’s a role for appealing to a higher power in any public organization. AA’s supporters see it as a scruffy grassroots organization of real people willing to get their hands dirty, who can cure addicts failed time and time again by a system of glitzy rehabs run by arrogant doctors who think their medical degrees make them better than people who have personally fought their own battles. Opponents see it as this awful cult that doesn’t provide any real treatment and just tells addicts that they’re terrible people who will never get better unless they sacrifice their identity to the collective.
As a result, the few sparks of light the research kindles are ignored, taken out of context, or misinterpreted.
The entire situation is complicated by a bigger question. We will soon find that AA usually does not work better or worse than various other substance abuse interventions. That leaves the sort of question that all those fancy-shmancy people with control groups in their studies don’t have to worry about – does anything work at all?
I.
We can start by just taking a big survey of people in Alcoholics Anonymous and seeing how they’re doing. On the one hand, we don’t have a control group. On the other hand…well, there really is no other hand, but people keep doing it.
According to AA’s own surveys, one-third of new members drop out by the end of their first month, half by the end of their third month, and three-quarters by the end of their first year. “Drop out” means they don’t go to AA meetings anymore, which could be for any reason including (if we’re feeling optimistic) them being so completely cured they no longer feel they need it.
There is an alternate reference going around that only 5% (rather than 25%) of AA members remain after their first year. This is a mistake caused by misinterpreting a graph showing that only five percent of members in their first year were in their twelfth month of membership, which is obviously completely different. Nevertheless, a large number of AA hate sites (and large rehabs!) cite the incorrect interpretation, for example the Orange Papers and RationalWiki’s page on Alcoholics Anonymous. In fact, just to keep things short, assume RationalWiki’s AA page makes every single mistake I warn against in the rest of this article, then use that to judge them in general. On the other hand, Wikipedia gets it right and I continue to encourage everyone to use it as one of the most reliable sources of medical information available to the public (I wish I was joking).
This retention information isn’t very helpful, since people can remain in AA without successfully quitting drinking, and people may successfully quit drinking without being in AA. However, various different sources suggest that, of people who stay in AA a reasonable amount of time, about half stop being alcoholic. These numbers can change wildly depending on how you define “reasonable amount of time” and “stop being alcoholic”. Here is a table, which I have cited on this blog before and will probably cite again:
Behold. Treatments that look very impressive (80% improved after six months!) turn out to be the same or worse as the control group. And comparing control group to control group, you can find that “no treatment” can appear to give wildly different outcomes (from 20% to 80% “recovery”) depending on what population you’re looking at and how you define “recovery”.
Twenty years ago, it was extremely edgy and taboo for a reputable scientist to claim that alcoholics could recover on their own. This has given way to the current status quo, in which pretty much everyone in the field writes journal articles all the time about how alcoholics can recover on their own, but make sure to harp upon how edgy and taboo they are for doing so. From these sorts of articles, we learn that about 80% of recovered alcoholics have gotten better without treatment, and many of them are currently able to drink moderately without immediately relapsing (something else it used to be extremely taboo to mention). Kate recently shared an good article about this: Most People With Addiction Simply Grow Out Of It: Why Is This Widely Denied?
Anyway, all this stuff about not being able to compare different populations, and the possibility of spontaneous recovery, just mean that we need controlled experiments. The largest number of these take a group of alcoholics, follow them closely, and then evaluate all of them – the AA-attending and the non-AA-attending – according to the same criteria. For example Morgenstern et al (1997), Humphreys et al (1997) and Moos (2006). Emrick et al (1993) is a meta-analyses of a hundred seventy three of these. All of these find that the alcoholics who end up going to AA meetings are much more likely to get better than those who don’t. So that’s good evidence the group is effective, right?
Bzzzt! No! Wrong! Selection bias!
People who want to quit drinking are more likely to go to AA than people who don’t want to quit drinking. People who want to quit drinking are more likely to actually quit drinking than those who don’t want to. This is a serious problem. Imagine if it is common wisdom that AA is the best, maybe the only, way to quit drinking. Then 100% of people who really want to quit would attend compared to 0% of people who didn’t want to quit. And suppose everyone who wants to quit succeeds, because secretly, quitting alcohol is really easy. Then 100% of AA members would quit, compared to 0% of non-members – the most striking result it is mathematically possible to have. And yet AA would not have made a smidgeon of difference.
But it’s worse than this, because attending AA isn’t just about wanting to quit. It’s also about having the resources to make it to AA. That is, wealthier people are more likely to hear about AA (better information networks, more likely to go to doctor or counselor who can recommend) and more likely to be able to attend AA (better access to transportation, more flexible job schedules). But wealthier people are also known to be better at quitting alcohol than poor people – either because the same positive personal qualities that helped them achieve success elsewhere help them in this battle as well, or just because they have fewer other stressors going on in their lives driving them to drink.
Finally, perseverance is a confounder. To go to AA, and to keep going for months and months, means you’ve got the willpower to drag yourself off the couch to do a potentially unpleasant thing. That’s probably the same willpower that helps you stay away from the bar.
And then there’s a confounder going the opposite direction. The worse your alcoholism is, the more likely you are to, as the organization itself puts it, “admit you have a problem”.
These sorts of longitudinal studies are almost useless and the field has mostly moved away from them. Nevertheless, if you look on the pro-AA sites, you will find them in droves, and all of them “prove” the organization’s effectiveness.
III.
It looks like we need randomized controlled trials. And we have them. Sort of.
Brandsma (1980)
is the study beloved of the AA hate groups, since it purports to show that people in Alcoholics Anonymous not only don’t get better, but are nine times more likely to binge drink than people who don’t go into AA at all.
There are a number of problems with this conclusion. First of all, if you actually look at the study, this is one of about fifty different findings. The other findings are things like “88% of treated subjects reported a reduction in drinking, compared to 50% of the untreated control group”.
Second of all, the increased binge drinking was significant at the 6 month followup period. It was not significant at the end of treatment, the 3 month followup period, the 9 month followup period, or the 12 month followup period. Remember, taking a single followup result out of the context of the other followup results is a classic piece of Dark Side Statistics and will send you to Science Hell.
Of multiple different endpoints, Alcoholics Anonymous did better than no treatment on almost all of them. It did worse than other treatments on some of them (dropout rates, binge drinking, MMPI scale) and the same as other treatments on others (abstinent days, total abstinence).
If you are pro-AA, you can say “Brandsma study proves AA works!”. If you are anti-AA, you can say “Brandsma study proves AA works worse than other treatments!”, although in practice most of these people prefer to quote extremely selective endpoints out of context.
However, most of the patients in the Brandsma study were people convicted of alcohol-related crimes ordered to attend treatment as part of their sentence. Advocates of AA make a good point that this population might be a bad fit for AA. They may not feel any personal motivation to treatment, which might be okay if you’re going to listen to a psychologist do therapy with you, but fatal for a self-help group. Since the whole point of AA is being in a community of like-minded individuals, if you don’t actually feel any personal connection to the project of quitting alcohol, it will just make you feel uncomfortable and out of place.
Also, uh, this just in, Brandsma didn’t use a real AA group, because the real AA groups make people be anonymous which makes it inconvenient to research stuff. He just sort of started his own non-anonymous group, let’s call it A, with no help from the rest of the fellowship, and had it do Alcoholics Anonymous-like stuff. On the other hand, many members of his control group went out into the community and…attended a real Alcoholics Anonymous, because Brandsma can’t exactly ethically tell them not to. So technically, there were more people in AA in the no-AA group than in the AA group. Without knowing more about Alcoholics Anonymous, I can’t know whether this objection is valid and whether Brandsma’s group did or didn’t capture the essence of the organization. Still, not the sort of thing you want to hear about a study.
Walsh et al (1991) is a similar study with similar confounders and similar results. Workers in an industrial plant who were in trouble for coming in drunk were randomly assigned either to an inpatient treatment program or to Alcoholics Anonymous. After a year of followup, 60% of the inpatient-treated workers had stayed sober, but only 30% of the AA-treated workers had.
The pro-AA side made three objections to this study, of which one is bad and two are good.
The bad objection was that AA is cheaper than hospitalization, so even if hospitalization is good, AA might be more efficient – after all, we can’t afford to hospitalize everyone. It’s a bad objection because the authors of the study did the math and found out that hospitalization was so much better than AA that it decreased the level of further medical treatment needed and saved the health system more money than it cost.
The first good objection: like the Brandsma study, this study uses people under coercion – in this case, workers who would lose their job if they refused. Fine.
The second good objection, and this one is really interesting: a lot of inpatient hospital rehab is AA. That is, when you go to an hospital for inpatient drug treatment, you attend AA groups every day, and when you leave, they make you keep going to the AA groups. In fact, the study says that “at the 12 month and 24 month assessments, the rates of AA affiliation and attendance in the past 6 months did not differ significantly among the groups.” Given that the hospital patients got hospital AA + regular AA, they were actually getting more AA than the AA group!
So all that this study proves is that AA + more AA + other things is better than AA. There was no “no AA” group, which makes it impossible to discuss how well AA does or doesn’t work. Frick.
Timko (2006) is the only study I can hesitantly half-endorse. This one has a sort of clever methodological trick to get around the limitation that doctors can’t ethically refuse to refer alcoholics to treatment. In this study, researchers at a Veterans’ Affairs hospital randomly assigned alcoholic patients to “referral” or “intensive referral”. In “referral”, the staff asked the patients to go to AA. In “intensive referral”, the researchers asked REALLY NICELY for the patients to go to AA, and gave them nice glossy brochures on how great AA was, and wouldn’t shut up about it, and arranged for them to meet people at their first AA meeting so they could have friends in AA, et cetera, et cetera. The hope was that more people in the “intensive referral” group would end out in AA, and
that indeed happenedscratch that, I just re-read the study and the same number of people in both groups went to AA and the intensive group actually completed a lower number of the 12 Steps on average, have I mentioned I hate all research and this entire field is terrible? But the intensive referral people were more likely to have “had a spiritual awakening” and “have a sponsor”, so it was decided the study wasn’t a complete loss and when it was found the intensive referral condition had slightly less alcohol use the authors decided to declare victory.So, whereas before we found that AA + More AA was better than AA, and that proved AA didn’t work, in this study we find that AA + More AA was better than AA, and that proves AA does work. You know, did I say I hesitantly half-endorsed this study? Scratch that. I hate this study too.
IV.
All right, @#%^ this $@!&*. We need a real study, everything all lined up in a row, none of this garbage. Let’s just hire half the substance abuse scientists in the country, throw a gigantic wad of money at them, give them as many patients as they need, let them take as long as they want, but barricade the doors of their office and not let them out until they’ve proven something important beyond a shadow of a doubt.
This was about how the scientific community felt in 1989, when they launched Project MATCH. This eight-year, $30 million dollar, multi-thousand patient trial was supposed to solve everything.
The people going into Project MATCH might have been a little overconfident. Maybe “not even Zeus could prevent this study from determining the optimal treatment for alcohol addiction” overconfident. This might have been a mistake.
The study was designed with three arms, one for each of the popular alcoholism treatments of the day. The first arm would be “twelve step facilitation”, a form of therapy based off of Alcoholics Anonymous. The second arm would be cognitive behavioral therapy, the most bog-standard psychotherapy in the world and one which by ancient tradition must be included in any kind of study like this. The third arm would be motivational enhancement therapy, which is a very short intervention where your doctor tells you all the reasons you should quit alcohol and tries to get you to convince yourself.
There wasn’t a “no treatment” arm. This is where the overconfidence might have come in. Everyone knew alcohol treatment worked. Surely you couldn’t dispute that. They just wanted to see which treatment worked best for which people. So you would enroll a bunch of different people – rich, poor, black, white, married, single, chronic alcoholic, new alcoholic, highly motivated, unmotivated – and see which of these people did best in which therapy. The result would be an algorithm for deciding where to send each of your patients. Rich black single chronic unmotivated alcoholic? We’ve found with p < 0.00001 that the best place for someone like that is in motivational enhancement therapy. Such was the dream. So, eight years and thirty million dollars and the careers of several prestigious researchers later, the results come in, and—yeah, everyone does exactly the same on every kind of therapy (with one minor, possibly coincidental exception). Awkward. “Everybody has won and all must have prizes!”. If you’re an optimist, you can say all treatments work and everyone can keep doing whatever they like best. If you’re a pessimist, you might start wondering whether anything works at all.
By my understanding this is also the confusing conclusion of Ferri, Amato & Davoli (2006), the Cochrane Collaboration’s attempt to get in on the AA action. Like all Cochrane Collaboration studies since the beginning of time, they find there is insufficient evidence to demonstrate the effectiveness of the intervention being investigated. This has been oft-quoted in the anti-AA literature. But by my reading, they had no control groups and were comparing AA to different types of treatment:
So the two best sources we have – Project MATCH and Cochrane – don’t find any significant differences between AA and other types of therapy. Now, to be fair, the inpatient treatment mentioned in Walsh et al wasn’t included, and inpatient treatment might be the gold standard here. But sticking to various forms of outpatient intervention, they all seem to be about the same.
So, the $64,000 question: do all of them work well, or do all of them work poorly?
V.
Alcoholism studies avoid control groups like they are on fire, presumably because it’s unethical not to give alcoholics treatment or something. However, there is one class of studies that doesn’t have that problem. These are the ones on “brief opportunistic intervention”, which is much like a turbocharged even shorter version of “motivational enhancement therapy”. Your doctor tells you ‘HELLO HAVE YOU CONSIDERED QUITTING ALCOHOL??!!’ and sees what happens.
Brief opportunistic intervention is the most trollish medical intervention ever, because here are all these brilliant psychologists and counselors trying to unravel the deepest mysteries of the human psyche in order to convince people to stop drinking, and then someone comes along and asks “Hey, have you tried just asking them politely?”. And it works.
Not consistently. But it works for about one in eight people. And the theory is that since it only takes a minute or two of a doctor’s time, it scales a lot faster than some sort of hideously complex hospital-based program that takes thousands of dollars and dozens of hours from everyone involved. If doctors would just spend five minutes with each alcoholic patient reminding them that no, really, alcoholism is really bad, we could cut the alcoholism rate by 1⁄8.
(this also works for smoking, by the way. I do this with every single one of my outpatients who smoke, and most of the time they roll their eyes, because their doctor is giving them that speech, but every so often one of them tells me that yeah, I’m right, they know they really should quit smoking and they’ll give it another try. I have never saved anyone’s life by dramatically removing their appendix at the last possible moment, but I have gotten enough patients to promise me they’ll try quitting smoking that I think I’ve saved at least one life just by obsessively doing brief interventions every chance I get. This is probably the most effective life-saving thing you can do as a doctor, enough so that if you understand it you may be licensed to ignore 80,000 Hours’ arguments on doctor replaceability)
Anyway, for some reason, it’s okay to do these studies with control groups. And they are so fast and easy to study that everyone studies them all the time. A meta-analysis of 19 studies is unequivocal that they definitely work.
Why do these work? My guess is that they do two things. First, they hit people who honestly didn’t realize they had a problem, and inform them that they do. Second, the doctor usually says they’ll “follow up on how they’re doing” the next appointment. This means that a respected authority figure is suddenly monitoring their drinking and will glare at them if they stay they’re still alcoholic. As someone who has gone into a panic because he has a dentist’s appointment in a week and he hasn’t been flossing enough – and then flossed until his teeth were bloody so the dentist wouldn’t be disappointed – I can sympathize with this.
But for our purposes, the brief opportunistic intervention sets a lower bound. It says “Here’s a really minimal thing that seems to work. Do other things work better than this?”
The “brief treatment” is the next step up from brief intervention. It’s an hour-or-so-long session (or sometimes a couple such sessions) with a doctor or counselor where they tell you some tips for staying off alcohol. I bring it up here because the brief treatment research community spends its time doing studies that show that brief treatments are just as good as much more intense treatments. This might be most comparable to the “motivational enhancement therapy” in the MATCH study.
Chapman and Huygens (1988) find that a single interview with a health professional is just as good as six weeks of inpatient treatment (I don’t know about their hospital in New Zealand, but for reference six weeks of inpatient treatment in my hospital costs about $40,000.)
Edwards (1977) finds that in a trial comparing “conventional inpatient or outpatient treatment complete with the full panoply of services available at a leading psychiatric institution and lasting several months” versus an hour with a doc, both groups do the same at one and two year followup.
And so on.
All of this is starting to make my head hurt, but it’s a familiar sort of hurt. It’s the way my head hurts when Scott Aaronson talks about complexity classes
. We have all of these different categories of things, and some of them are the same as others and others are bigger than others but we’re not sure exactly where all of them stand.
We have classes “no treatment”, “brief opportunistic intervention”, “brief treatment”, “Alcoholics Anonymous”, “psychotherapy”, and “inpatient”.
We can prove that BOI > NT, and that AA = PT. Also that BT = IP = PT. We also have that IP > AA, which unfortunately we can use to prove a contradiction, so let’s throw it out for now.
So the hierarchy of classes seems to be (NT) < (BOI) ? (BT, IP, AA, PT) - in other words, no treatment is the worst, brief opportunistic intervention is better, and then somewhere in there we have this class of everything else that is the same.
Can we prove that BOI = BT?
We have some good evidence for this, once again from our Handbook. A study in Edinburgh finds that five minutes of psychiatrist advice (brief opportunistic intervention) does the same as sixty minutes of advice plus motivational interviewing (brief treatment).
So if we take all this seriously, then it looks like every psychosocial treatment (including brief opportunistic intervention) is the same, and all are better than no treatment. This is a common finding in psychiatry and psychology – for example, all common antidepressants are better than no treatment but work about equally well; all psychotherapies are better than no treatment but work about equally well, et cetera. It’s still an open question what this says about our science and our medicine.
The strongest counterexample to this is Walsh et al which finds the inpatient hospital stay works better than the AA referral, but this study looks kind of lonely compared to the evidence on the other side. And even the authors admit they were surprised by the effectiveness of the hospital there.
And let’s go back to Project MATCH. There wasn’t a control group. But there were the people who dropped out of the study, who said they’d go to AA or psychotherapy but never got around to it. Cutter and Fishbain (2005) take a look at what happened to these folks. They find that the dropouts did 75% as well as the people in any of the therapy groups, and that most of the effect of the therapy groups occurred in the first week (ie people dropped out after one week did about 95% as well as people who stayed in).
To me this suggests two things. First, therapy is only a little helpful over most people quitting on their own. Second, insofar as therapy is helpful, the tiniest brush with therapy is enough to make someone think “Okay, I’ve had some therapy, I’ll be better now”. Just like with the brief opportunistic interventions, five minutes of almost anything is enough.
This is a weird conclusion, but I think it’s the one supported by the data.
VI.
I should include a brief word about this giant table.
I see it everywhere. It looks very authoritative and impressive and, of course, giant. I believe the source is Miller’s Handbook of Alcoholism Treatment Approaches: Effective Alternatives, 3rd Edition
, the author of which is known as a very careful scholar whom I cannot help but respect.
And the table does a good thing in discussing medications like acamprosate and naltrexone, which are very important and effective interventions but which will not otherwise be showing up in this post.
However, the therapy part of the table looks really wrong to me.
First of all, I notice acupuncture is ranked 17 out of 48, putting in a much, much better showing than treatments like psychotherapy, counseling, or education. Seems fishy.
Second of all, I notice that motivational enhancement (#2), cognitive therapy (#13), and twelve-step (#37) are all about as far apart as could be, but the largest and most powerful trial ever, Project MATCH, found all three to be about equal in effectiveness.
Third of all, I notice that cognitive therapy is at #13, but psychotherapy is at #46. But cognitive therapy is a kind of psychotherapy.
Fourth of all, I notice that brief interventions, motivational enhancement, confrontational counseling, psychotherapy, general alcoholism counseling, and education are all over. But a lot of these are hard to differentiate from one another.
The table seems messed up to me. Part of it is because it is about evidence base rather than effectiveness (consider that handguns have a stronger evidence base than the atomic bomb, since they have been used many more times in much better controlled conditions, but the atomic bomb is more effective) and therefore acupuncture, which is poorly studied, can rank quite high compared to things which have even one negative study.
But part of it just seems wrong. I haven’t read the full book, but I blame the tendency to conflate studies showing “X does not work better than anything else” with “X does not work”.
Remember, whenever there are meta-analyses that contradict single very large well-run studies, go with the single very large well-run study, especially when the meta-analysis is as weird as this one. Project MATCH is the single very large well-run study, and it says this is balderdash. I’m guessing it’s trying to use some weird algorithmic methodology to automatically rate and judge each study, but that’s no substitute for careful human review.
VII.
In conclusion, as best I can tell – and it is not very well, because the studies that could really prove anything robustly haven’t been done – most alcoholics get better on their own. All treatments for alcoholism, including Alcoholics Anonymous, psychotherapy, and just a few minutes with a doctor explaining why she thinks you need to quit, increase this already-high chance of recovery a small but nonzero amount. Furthermore, they are equally effective after only a tiny dose: your first couple of meetings, your first therapy session. Some studies suggest that inpatient treatment with outpatient followup may be better than outpatient treatment alone, but other studies contradict this and I am not confident in the assumption.
So does Alcoholics Anonymous work? Though I cannot say anything authoritatively, my impression is: Yes, but only a tiny bit, and for many people five minutes with a doctor may work just as well as years completing the twelve steps. As such, individual alcoholics may want to consider attending if they don’t have easier options; doctors might be better off just talking to their patients themselves.
If this is true – and right now I don’t have much confidence that it is, it’s just a direction that weak and contradictory data are pointing – it would be really awkward for the multibazillion-dollar treatment industry.
More worrying, I am afraid of what it would do to the War On Drugs. Right now one of the rallying cries for the anti-Drug-War movement is “treatment, not prison”. And although I haven’t looked seriously at the data for any drug besides alcohol. I think some data there are similar. There’s very good medication for drugs – for example methadone and suboxone for opiate abuse – but in terms of psychotherapy it’s mostly the same stuff you get for alcohol. Rehabs, whether they work or not, seem to serve an important sort of ritual function, where if you can send a drug abuser to a rehab you at least feel like something has been done. Deny people that ritual, and it might make prison the only politically acceptable option.
In terms of things to actually treat alcoholism, I remain enamoured of the Sinclair Method, which has done crazy outrageous stuff like conduct an experiment with an actual control group. But I haven’t investigated enough to know whether my early excitement about them looks likely to pan out or not.
I would not recommend quitting any form of alcohol treatment that works for you, or refusing to try a form of treatment your doctor recommends, based on any of this information.