Against Devil’s Advocacy

Eliezer Yudkowsky9 Jun 2008 4:15 UTC
50 points
60 comments6 min readLW linkArchive

From an article by Michael Ruse:

Richard Dawkins once called me a “creep.” He did so very publicly but meant no personal offense, and I took none: We were, and still are, friends. The cause of his ire—his anguish, even—was that, in the course of a public discussion, I was defending a position I did not truly hold. We philosophers are always doing this; it’s a version of the reductio ad absurdum argument. We do so partly to stimulate debate (especially in the classroom), partly to see how far a position can be pushed before it collapses (and why the collapse), and partly (let us be frank) out of sheer bloody-mindedness, because we like to rile the opposition.

Dawkins, however, has the moral purity—some would say the moral rigidity—of the evangelical Christian or the committed feminist. Not even for the sake of argument can he endorse something that he thinks false. To do so is not just mistaken, he feels; in some deep sense, it is wrong. Life is serious, and there are evils to be fought. There must be no compromise or equivocation, even for pedagogical reasons. As the Quakers say, “Let your yea be yea, and your nay, nay.”

Michael Ruse doesn’t get it.

When I was a kid and my father was teaching me about skepticism -

(Dad was an avid skeptic and Martin Gardner /​ James Randi fan, as well as being an Orthodox Jew. Let that be a lesson on the anti-healing power of compartmentalization.)

- he used the example of the hypothesis: “There is an object one foot across in the asteroid belt composed entirely of chocolate cake.” You would have to search the whole asteroid belt to disprove this hypothesis. But though this hypothesis is very hard to disprove, there aren’t good arguments for it.

And the child-Eliezer asked his mind to search for arguments that there was a chocolate cake in the asteroid belt. Lo, his mind returned the reply: “Since the asteroid-belt-chocolate-cake is one of the classic examples of a bad hypothesis, if anyone ever invents a time machine, some prankster will toss a chocolate cake back into the 20th-century asteroid belt, making it true all along.”

Thus—at a very young age—I discovered that my mind could, if asked, invent arguments for anything.

I know people whose sanity has been destroyed by this discovery. They conclude that Reason can be used to argue for anything. And so there is no point in arguing that God doesn’t exist, because you could just as well argue that God does exist. Nothing left but to believe whatever you want.

Having given up, they develop whole philosophies of self-inflation to make their despair seem Deeply Wise. If they catch you trying to use Reason, they will smile and pat you on the head and say, “Oh, someday you’ll discover that you can argue for anything.”

Perhaps even now, my readers are thinking, “Uh-oh, Eliezer can rationalize anything, that’s not a good sign.”

But you know… being mentally agile doesn’t always doom you to disaster. I mean, you might expect that it would. Yet sometimes practice turns out to be different from theory.

Rationalization came too easily to me. It was visibly just a game.

If I had been less imaginative and more easily stumped—if I had not found myself able to argue for any proposition no matter how bizarre—then perhaps I would have confused the activity with thinking.

But I could argue even for chocolate cake in the asteroid belt. It wasn’t even difficult; my mind coughed up the argument immediately. It was very clear that this was fake thinking and not real thinking. I never for a moment confused the game with real life. I didn’t start thinking there might really be a chocolate cake in the asteroid belt.

You might expect that any child with enough mental agility to come up with arguments for anything, would surely be doomed. But intelligence doesn’t always do so much damage as you might think. In this case, it just set me up, at a very early age, to distinguish “reasoning” from “rationalizing”. They felt different.

Perhaps I’m misremembering… but it seems to me that, even at that young age, I looked at my mind’s amazing clever argument for a time-traveling chocolate cake, and thought: I’ve got to avoid doing that.

(Though there are much more subtle cognitive implementations of rationalizing processes, than blatant, obvious, conscious search for favorable arguments. A wordless flinch away from an idea can undo you as surely as a deliberate search for arguments against it. Those subtler processes, I only began to notice years later.)

I picked up an intuitive sense that real thinking was that which could force you into an answer whether you liked it or not, and fake thinking was that which could argue for anything.

This was an incredibly valuable lesson -

(Though, like many principles that my young self obtained by reversing stupidity, it gave good advice on specific concrete problems; but went wildly astray when I tried to use it to make abstract deductions, e.g. about the nature of morality.)

- which was one of the major drivers behind my break with Judaism. The elaborate arguments and counterarguments of ancient rabbis, looked like the kind of fake thinking I did to argue that there was chocolate cake in the asteroid belt. Only the rabbis had forgotten it was a game, and were actually taking it seriously.

Believe me, I understand the Traditional argument behind Devil’s Advocacy. By arguing the opposing position, you increase your mental flexibility. You shake yourself out of your old shoes. You get a chance to gather evidence against your position, instead of arguing for it. You rotate things around, see them from a different viewpoint. Turnabout is fair play, so you turn about, to play fair.

Perhaps this is what Michael Rose was thinking, when he accused Richard Dawkins of “moral rigidity”.

I surely don’t mean to teach people to say: “Since I believe in fairies, I ought not to expect to find any good arguments against their existence, therefore I will not search because the mental effort has a low expected utility.” That comes under the heading of: If you want to shoot your foot off, it is never the least bit difficult to do so.

Maybe there are some stages of life, or some states of mind, in which you can be helped by trying to play Devil’s Advocate. Students who have genuinely never thought of trying to search for arguments on both sides of an issue, may be helped by the notion of “Devil’s Advocate”.

But with anyone in this state of mind, I would sooner begin by teaching them that policy debates should not appear one-sided. There is no expectation against having strong arguments on both sides of a policy debate; single actions have multiple consequences. If you can’t think of strong arguments against your most precious favored policies, or strong arguments for policies that you hate but which other people endorse, then indeed, you very likely have a problem that could be described as “failing to see the other points of view”.

You, dear reader, are probably a sophisticated enough reasoner that if you manage to get yourself stuck in an advanced rut, dutifully playing Devil’s Advocate won’t get you out of it. You’ll just subconsciously avoid any Devil’s arguments that make you genuinely nervous, and then congratulate yourself for doing your duty. People at this level need stronger medicine. (So far I’ve only covered medium-strength medicine.)

If you can bring yourself to a state of real doubt and genuine curiosity, there is no need for Devil’s Advocacy. You can investigate the contrary position because you think it might be really genuinely true, not because you are playing games with time-traveling chocolate cakes. If you cannot find this trace of true doubt within yourself, can merely playing Devil’s Advocate help you?

I have no trouble thinking of arguments for why the Singularity won’t happen for another 50 years. With some effort, I can make a case for why it might not happen in 100 years. I can also think of plausible-sounding scenarios in which the Singularity happens in two minutes, i.e., someone ran a covert research project and it is finishing right now. I can think of plausible arguments for 10-year, 20-year, 30-year, and 40-year timeframes.

This is not because I am good at playing Devil’s Advocate and coming up with clever arguments. It’s because I really don’t know. A true doubt exists in each case, and I can follow my doubt to find the source of a genuine argument. Or if you prefer: I really don’t know, because I can come up with all these plausible arguments.

On the other hand, it is really hard for me to visualize the proposition that there is no kind of mind substantially stronger than a human one. I have trouble believing that the human brain, which just barely suffices to run a technological civilization that can build a computer, is also the theoretical upper limit of effective intelligence. I cannot argue effectively for that, because I do not believe it. Or if you prefer, I do not believe it, because I cannot argue effectively for it. If you want that idea argued, find someone who really believes it. Since a very young age, I’ve been endeavoring to get away from those modes of thought where you can argue for just anything.

In the state of mind and stage of life where you are trying to distinguish rationality from rationalization, and trying to tell the difference between weak arguments and strong arguments, Devil’s Advocate cannot lead you to unfake modes of reasoning. Its only power is that it may perhaps show you the fake modes which operate equally well on any side, and tell you when you are uncertain.

There is no chess grandmaster who can play only black, or only white; but in the battles of Reason, a soldier who fights with equal strength on any side has zero force.

That’s what Richard Dawkins understands that Michael Ruse doesn’t—that Reason is not a game.

Added: Brandon argues that Devil’s Advocacy is most importantly a social rather than individual process, which aspect I confess I wasn’t thinking about.

Eliezer Yudkowsky9 Jun 2008 4:15 UTC
50 points
60 comments6 min readLW linkArchive