Schools Proliferating Without Evidence

Eliezer Yudkowsky15 Mar 2009 6:43 UTC
64 points
58 comments3 min readLW linkArchive

Robyn Dawes, author of one of the original papers from Judgment Under Uncertainty and of the book Rational Choice in an Uncertain World—one of the few who tries really hard to import the results to real life—is also the author of House of Cards: Psychology and Psychotherapy Built on Myth.

From House of Cards, chapter 1:

The ability of these professionals has been subjected to empirical scrutiny—for example, their effectiveness as therapists (Chapter 2), their insight about people (Chapter 3), and the relationship between how well they function and the amount of experience they have had in their field (Chapter 4). Virtually all the research—and this book will reference more than three hundred empirical investigations and summaries of investigations—has found that these professionals’ claims to superior intuitive insight, understanding, and skill as therapists are simply invalid...

Remember Rorschach ink-blot tests? It’s such an appealing argument: the patient looks at the ink-blot and says what he sees, the psychotherapist interprets their psychological state based on this. There’ve been hundreds of experiments looking for some evidence that it actually works. Since you’re reading this, you can guess the answer is simply “No.” Yet the Rorschach is still in use. It’s just such a good story that psychotherapists just can’t bring themselves to believe the vast mounds of experimental evidence saying it doesn’t work—

—which tells you what sort of field we’re dealing with here.

And the experimental results on the field as a whole are commensurate. Yes, patients who see psychotherapists have been known to get better faster than patients who simply do nothing. But there is no statistically discernible difference between the many schools of psychotherapy. There is no discernible gain from years of expertise.

And there’s also no discernible difference between seeing a psychotherapist and spending the same amount of time talking to a randomly selected college professor from another field. It’s just talking to anyone that helps you get better, apparently.

In the entire absence of the slightest experimental evidence for their effectiveness, psychotherapists became licensed by states, their testimony accepted in court, their teaching schools accredited, and their bills paid by health insurance.

And there was also a huge proliferation of “schools”, of traditions of practice, in psychotherapy; despite—or perhaps because of—the lack of any experiments showing that one school was better than another...

I should really post more some other time on all the sad things this says about our world; about how the essence of medicine, as recognized by society and the courts, is not a repertoire of procedures with statistical evidence for their healing effectiveness; but, rather, the right air of authority.

But the subject today is the proliferation of traditions in psychotherapy. So far as I can discern, this was the way you picked up prestige in the field—not by discovering an amazing new technique whose effectiveness could be experimentally verified and adopted by all; but, rather, by splitting off your own “school”, supported by your charisma as founder, and by the good stories you told about all the reasons your techniques should work.

This was probably, to no small extent, responsible for the existence and continuation of psychotherapy in the first place—the promise of making yourself a Master, like Freud who’d done it first (also without the slightest scrap of experimental evidence). That’s the brass ring of success to chase—the prospect of being a guru and having your own adherents. It’s the struggle for adherents that keeps the clergy vital.

That’s what happens to a field when it unbinds itself from the experimental evidence—though there were other factors that also placed psychotherapists at risk, such as the deference shown them by their patients, the wish of society to believe that mental healing was possible, and, of course, the general dangers of telling people how to think.

The field of hedonic psychology (happiness studies) began, to some extent, with the realization that you could measure happiness—that there was a family of measures that by golly did validate well against each other.

The act of creating a new measurement creates new science; if it’s a good measurement, you get good science.

If you’re going to create an organized practice of anything, you really do need some way of telling how well you’re doing, and a practice of doing serious testing—that means a control group, an experimental group, and statistics—on plausible-sounding techniques that people come up with. You really need it.

Added: Dawes wrote in the 80s and I know that the Rorschach was still in use as recently as the 90s, but it’s possible matters have improved since then (as one commenter states). I do remember hearing that there was positive evidence for the greater effectiveness of cognitive-behavioral therapy.

Eliezer Yudkowsky15 Mar 2009 6:43 UTC
64 points
58 comments3 min readLW linkArchive