Pre­dictably Wrong

This, the first book of “Rationality: AI to Zombies” (also known as “The Sequences”), begins with cognitive bias. The rest of the book won’t stick to just this topic; bad habits and bad ideas matter, even when they arise from our minds’ contents as opposed to our minds’ structure.

It is cognitive bias, however, that provides the clearest and most direct glimpse into the stuff of our psychology, into the shape of our heuristics and the logic of our limitations. It is with bias that we will begin.

Fake Beliefs

An account of irrationality would be incomplete if it provided no theory about how rationality works—or if its “theory” only consisted of vague truisms, with no precise explanatory mechanism. This sequence asks why it’s useful to base one’s behavior on “rational” expectations, and what it feels like to do so.

Notic­ing Confusion

Mys­te­ri­ous Answers

This sequence asks whether science resolves the problems raised so far. Scientists base their models on repeatable experiments, not speculation or hearsay. And science has an excellent track record compared to anecdote, religion, and . . . pretty much everything else. Do we still need to worry about “fake” beliefs, confirmation bias, hindsight bias, and the like when we’re working with a community of people who want to explain phenomena, not just tell appealing stories?

Overly Con­ve­nient Excuses

This sequence focuses on questions that are as probabilistically clear-cut as questions get. The Bayes-optimal answer is often infeasible to compute, but errors like confirmation bias can take root even in cases where the available evidence is overwhelming and we have plenty of time to think things over.

Poli­tics and Rationality

Now we move into a murkier area. Mainstream national politics, as debated by TV pundits, is famous for its angry, unproductive discussions. On the face of it, there’s something surprising about that. Why do we take political disagreements so personally, even when the machinery and effects of national politics are so distant from us in space or in time? For that matter, why do we not become more careful and rigorous with the evidence when we’re dealing with issues we deem important?

Against Rationalization

The last sequence focused in on how feeling tribal often distorts our ability to reason. Now we’ll explore one particular cognitive mechanism that causes this: much of our reasoning process is really rationalization—story-telling that makes our current beliefs feel more coherent and justified, without necessarily improving their accuracy.

Against Doublethink

This short sequence explores another cognitive pattern that hinders our ability to update on evidence: George Orwell’s ‘doublethink’ - the attempt to deceive oneself.

See­ing with Fresh Eyes

On the challenge of recognizing evidence that doesn’t fit our expectations and assumptions.

Death Spirals

Leveling up in rationality means encountering a lot of interesting and powerful new ideas. In many cases, it also means making friends who you can bounce ideas off of and finding communities that encourage you to better yourself. This sequence discusses some important hazards that can afflict groups united around common interests and amazing shiny ideas, which will need to be overcome if we’re to get the full benefits out of rationalist communities.

Let­ting Go

Our natural state isn’t to change our minds like a Bayesian would. Getting the people in opposing tribes to notice what they’re really seeing won’t be as easy as reciting the axioms of probability theory to them. As Luke Muehlhauser writes, in The Power of Agency:

You are not a Bayesian homunculus whose reasoning is “corrupted” by cognitive biases.

You just are cognitive biases.

Confirmation bias, status quo bias, correspondence bias, and the like are not tacked on to our reasoning; they are its very substance.

That doesn’t mean that debiasing is impossible. We aren’t perfect calculators underneath all our arithmetic errors, either. Many of our mathematical limitations result from very deep facts about how the human brain works. Yet we can train our mathematical abilities; we can learn when to trust and distrust our mathematical intuitions, and share our knowledge, and help one another; we can shape our environments to make things easier on us, and build tools to offload much of the work.

Our biases are part of us. But there is a shadow of Bayesianism present in us as well, a flawed apparatus that really can bring us closer to truth. No homunculus—but still, some truth. Enough, perhaps, to get started.

The Sim­ple Math of Evolution

The first sequence of The Machine in the Ghost aims to communicate the dissonance and divergence between our hereditary history, our present-day biology, and our ultimate aspirations. This will require digging deeper than is common in introductions to evolution for non-biologists, which often restrict their attention to surface-level features of natural selection.

Frag­ile Purposes

This sequence abstracts from human cognition and evolution to the idea of minds and goal-directed systems at their most general. These essays serve the secondary purpose of explaining the author’s general approach to philosophy and the science of rationality, which is strongly informed by his work in AI.

A Hu­man’s Guide to Words

This sequence discusses the basic relationship between cognition and concept formation. 37 Ways That Words Can Be Wrong is a guide to the sequence.

Lawful Truth

Just as it was useful to contrast humans as goal-oriented systems with inhuman processes in evolutionary biology and artificial intelligence, it will be useful in the coming sequences of essays to contrast humans as physical systems with inhuman processes that aren’t mind-like.

We humans are, after all, built out of inhuman parts. The world of atoms looks nothing like the world as we ordinarily think of it, and certainly looks nothing like the world’s conscious denizens as we ordinarily think of them. As Giulio Giorello put the point in an interview with Daniel Dennett: “Yes, we have a soul. But it’s made of lots of tiny robots.”

We start with a sequence on the basic links between physics and human cognition.

Re­duc­tion­ism 101

Joy in the Merely Real

..Do not all charms fly

At the mere touch of cold philosophy?

There was an awful rainbow once in heaven:

We know her woof, her texture; she is given

In the dull catalogue of common things.

—John Keats, Lamia

Phys­i­cal­ism 201

Can we ever know what it’s like to be a bat? Traditional dualism, with its immaterial souls freely floating around violating physical laws, may be false; but what about the weaker thesis, that consciousness is a “further fact” not fully explainable by the physical facts? A number of philosophers and scientists have found this line of reasoning persuasive. If we feel this argument’s intuitive force, should we grant its conclusion and ditch physicalism?

We certainly shouldn’t reject it just because it sounds strange or feels vaguely unscientific. But how does the argument stand up to a technical understanding of how explanation and belief work? Are there any hints we can take from the history of science, or from our understanding of the physical mechanisms underlying evidence? This is the question that this sequence will attempt to answer.

Quan­tum Physics and Many Worlds

Quantum mechanics is our best mathematical model of the universe to date, powerfully confirmed by a century of tests. However, interpreting what the experimental results mean—how and when the Schrödinger equation and Born’s rule interact—is a topic of much contention, with the main disagreement being between the Everett and the Copenhagen interpretations.

Yudkowsky uses this scientific controversy as a proving ground for some central ideas from previous sequences: map-territory distinctions, mysterious answers, Bayesianism, and Occam’s Razor.

Science and Rationality

The final sequence in this book tie these ideas together, and draws some conclusions on the strength of our scientific institutions.

Fake Preferences

On failed attempts at theories of value.

Value Theory

On obstacles to developing a new theory, and some intuitively desirable features of such a theory.

Quan­tified Humanism

On the tricky question of how we should apply such theories to our ordinary moral intuitions and decision-making.

Yud­kowsky’s Com­ing of Age

This sequence provides a last in-depth illustration of the dynamics of irrational belief, this time spotlighting the author’s own intellectual history.

Challeng­ing the Difficult

This sequences asks what it takes to solve a truly difficult problem—including demands that go beyond epistemic rationality.

The Craft and the Community

Discusses rationality groups and group rationality, raising the questions:

• Can rationality be learned and taught?

• If so, how much improvement is possible?

How can we be confident we’re seeing a real effect in a rationality intervention, and picking out the right cause?

• What community norms would make this process of bettering ourselves easier?

• Can we effectively collaborate on large-scale problems without sacrificing our freedom of thought and conduct?

Above all: What’s missing? What should be in the next generation of rationality primers—the ones that replace this text, improve on its style, test its prescriptions, supplement its content, and branch out in altogether new directions?