Rationality Reading Group: Part Z: The Craft and the Community
This is part of a semi-monthly reading group on Eliezer Yudkowsky’s ebook, Rationality: From AI to Zombies. For more information about the group, see the announcement post.
Welcome to the Rationality reading group. This fortnight we discuss Part Z: The Craft and the Community (pp. 1651-1750). This post summarizes each article of the sequence, linking to the original LessWrong post where available.
Z. The Craft and the Community
312. Raising the Sanity Waterline—Behind every particular failure of social rationality is a larger and more general failure of social rationality; even if all religious content were deleted tomorrow from all human minds, the larger failures that permit religion would still be present. Religion may serve the function of an asphyxiated canary in a coal mine—getting rid of the canary doesn’t get rid of the gas. Even a complete social victory for atheism would only be the beginning of the real work of rationalists. What could you teach people without ever explicitly mentioning religion, that would raise their general epistemic waterline to the point that religion went underwater?
313. A Sense That More Is Possible—The art of human rationality may have not been much developed because its practitioners lack a sense that vastly more is possible. The level of expertise that most rationalists strive to develop is not on a par with the skills of a professional mathematician—more like that of a strong casual amateur. Self-proclaimed “rationalists” don’t seem to get huge amounts of personal mileage out of their craft, and no one sees a problem with this. Yet rationalists get less systematic training in a less systematic context than a first-dan black belt gets in hitting people.
314. Epistemic Viciousness—An essay by Gillian Russell on “Epistemic Viciousness in the Martial Arts” generalizes amazingly to possible and actual problems with building a community around rationality. Most notably the extreme dangers associated with “data poverty”—the difficulty of testing the skills in the real world. But also such factors as the sacredness of the dojo, the investment in teachings long-practiced, the difficulty of book learning that leads into the need to trust a teacher, deference to historical masters, and above all, living in data poverty while continuing to act as if the luxury of trust is possible.
315. Schools Proliferating Without Evidence—The branching schools of “psychotherapy”, another domain in which experimental verification was weak (nonexistent, actually), show that an aspiring craft lives or dies by the degree to which it can be tested in the real world. In the absence of that testing, one becomes prestigious by inventing yet another school and having students, rather than excelling at any visible performance criterion. The field of hedonic psychology (happiness studies) began, to some extent, with the realization that you could measure happiness—that there was a family of measures that by golly did validate well against each other. The act of creating a new measurement creates new science; if it’s a good measurement, you get good science.
316. Three Levels of Rationality Verification—How far the craft of rationality can be taken, depends largely on what methods can be invented for verifying it. Tests seem usefully stratifiable into reputational, experimental, andorganizational. A “reputational” test is some real-world problem that tests the ability of a teacher or a school (like running a hedge fund, say) - “keeping it real”, but without being able to break down exactly what was responsible for success. An “experimental” test is one that can be run on each of a hundred students (such as a well-validated survey). An “organizational” test is one that can be used to preserve the integrity of organizations by validating individuals or small groups, even in the face of strong incentives to game the test. The strength of solution invented at each level will determine how far the craft of rationality can go in the real world.
317. Why Our Kind Can’t Cooperate—The atheist/libertarian/technophile/sf-fan/early-adopter/programmer/etc crowd, aka “the nonconformist cluster”, seems to be stunningly bad at coordinating group projects. There are a number of reasons for this, but one of them is that people are as reluctant to speak agreement out loud, as they are eager to voice disagreements—the exact opposite of the situation that obtains in more cohesive and powerful communities. This is not rational either! It is dangerous to be half a rationalist (in general), and this also applies to teaching only disagreement but not agreement, or only lonely defiance but not coordination. The pseudo-rationalist taboo against expressing strong feelings probably doesn’t help either.
318. Tolerate Tolerance—One of the likely characteristics of someone who sets out to be a “rationalist” is a lower-than-usual tolerance for flawed thinking. This makes it very important to tolerate other people’s tolerance—to avoid rejecting them because they tolerate people you wouldn’t—since otherwise we must all have exactly the same standards of tolerance in order to work together, which is unlikely. Even if someone has a nice word to say about complete lunatics and crackpots—so long as they don’t literally believe the same ideas themselves—try to be nice to them? Intolerance of tolerance corresponds to punishment of non-punishers, a very dangerous game-theoretic idiom that can lock completely arbitrary systems in place even when they benefit no one at all.
319. Your Price for Joining—The game-theoretical puzzle of the Ultimatum game has its reflection in a real-world dilemma: How much do you demand that an existing group adjust toward you, before you will adjust toward it? Our hunter-gatherer instincts will be tuned to groups of 40 with very minimal administrative demands and equal participation, meaning that we underestimate the inertia of larger and more specialized groups and demand too much before joining them. In other groups this resistance can be overcome by affective death spirals and conformity, but rationalists think themselves too good for this—with the result that people in the nonconformist cluster often set their joining prices way way way too high, like an 50-way split with each player demanding 20% of the money. Nonconformists need to move in the direction of joining groups more easily, even in the face of annoyances and apparent unresponsiveness. If an issue isn’t worth personally fixing by however much effort it takes, it’s not worth a refusal to contribute.
320. Can Humanism Match Religion’s Output? - Anyone with a simple and obvious charitable project—responding with food and shelter to a tidal wave in Thailand, say—would be better off by far pleading with the Pope to mobilize the Catholics, rather than with Richard Dawkins to mobilize the atheists. For so long as this is true, any increase in atheism at the expense of Catholicism will be something of a hollow victory, regardless of all other benefits. Can no rationalist match the motivation that comes from the irrational fear of Hell? Or does the real story have more to do with the motivating power of physically meeting others who share your cause, and group norms of participating?
321. Church vs. Taskforce—Churches serve a role of providing community—but they aren’t explicitly optimized for this, because their nominal role is different. If we desire community without church, can we go one better in the course of deleting religion? There’s a great deal of work to be done in the world; rationalist communities might potentially organize themselves around good causes, while explicitly optimizing for community.
322. Rationality: Common Interest of Many Causes—Many causes benefit particularly from the spread of rationality—because it takes a little more rationality than usual to see their case, as a supporter, or even just a supportive bystander. Not just the obvious causes like atheism, but things like marijuana legalization. In the case of my own work this effect was strong enough that after years of bogging down I threw up my hands and explicitly recursed on creating rationalists. If such causes can come to terms with not individually capturing all the rationalists they create, then they can mutually benefit from mutual effort on creating rationalists. This cooperation may require learning to shut up about disagreements between such causes, and not fight over priorities, except in specialized venues clearly marked.
323. Helpless Individuals—When you consider that our grouping instincts are optimized for 50-person hunter-gatherer bands where everyone knows everyone else, it begins to seem miraculous that modern-day large institutions survive at all. And in fact, the vast majority of large modern-day institutions simply fail to exist in the first place. This is why funding of Science is largely through money thrown at Science rather than donations from individuals—research isn’t a good emotional fit for the rare problems that individuals can manage to coordinate on. In fact very few things are, which is why e.g. 200 million adult Americans have such tremendous trouble supervising the 535 members of Congress. Modern humanity manages to put forth very little in the way of coordinated individual effort to serve our collective individual interests.
324. Money: The Unit of Caring—Omohundro’s resource balance principle implies that the inside of any approximately rational system has a common currency of expected utilons. In our world, this common currency is called “money” and it is the unit of how much society cares about something—a brutal yet obvious point. Many people, seeing a good cause, would prefer to help it by donating a few volunteer hours. But this avoids the tremendous gains of comparative advantage, professional specialization, and economies of scale—the reason we’re not still in caves, the only way anything ever gets done in this world, the tools grownups use when anyone really cares. Donating hours worked within a professional specialty and paying-customer priority, whether directly, or by donating the money earned to hire other professional specialists, is far more effective than volunteering unskilled hours.
325. Purchase Fuzzies and Utilons Separately—Wealthy philanthropists typically make the mistake of trying to purchase warm fuzzy feelings, status among friends, and actual utilitarian gains, simultaneously; this results in vague pushes along all three dimensions and a mediocre final result. It should be far more effective to spend some money/effort on buying altruistic fuzzies at maximum optimized efficiency (e.g. by helping people in person and seeing the results in person), buying status at maximum efficiency (e.g. by donating to something sexy that you can brag about, regardless of effectiveness), and spending most of your money on expected utilons (chosen through sheer cold-blooded shut-up-and-multiply calculation, without worrying about status or fuzzies).
326. Bystander Apathy—The bystander effect is when groups of people are less likely to take action than an individual. There are a few explanations for why this might be the case.
327. Collective Apathy and the Internet—The causes of bystander apathy are even worse on the Internet. There may be an opportunity here for a startup to deliberately try to avert bystander apathy in online group coordination.
328. Incremental Progress and the Valley—The optimality theorems for probability theory and decision theory, are for perfect probability theory and decision theory. There is no theorem that incremental changes toward the ideal, starting from a flawed initial form, must yield incremental progress at each step along the way. Since perfection is unattainable, why dare to try for improvement? But my limited experience with specialized applications suggests that given enough progress, one can achieve huge improvements over baseline—it just takes a lot of progress to get there.
329. Bayesians vs. Barbarians—Suppose that a country of rationalists is attacked by a country of Evil Barbarians who know nothing of probability theory or decision theory. There’s a certain concept of “rationality” which says that the rationalists inevitably lose, because the Barbarians believe in a heavenly afterlife if they die in battle, while the rationalists would all individually prefer to stay out of harm’s way. So the rationalist civilization is doomed; it is too elegant and civilized to fight the savage Barbarians… And then there’s the idea that rationalists should be able to (a) solve group coordination problems, (b) care a lot about other people and (c) win...
330. Beware of Other-Optimizing—Aspiring rationalists often vastly overestimate their own ability to optimize other people’s lives. They read nineteen webpages offering productivity advice that doesn’t work for them… and then encounter the twentieth page, or invent a new method themselves, and wow, it really works—they’ve discovered the true method. Actually, they’ve just discovered the one method in twenty that works for them, and their confident advice is no better than randomly selecting one of the twenty blog posts. Other-Optimizing is exceptionally dangerous when you have power over the other person—for then you’ll just believe that they aren’t trying hard enough.
331. Practical Advice Backed by Deep Theories—Practical advice is genuinely much, much more useful when it’s backed up by concrete experimental results, causal models that are actually true, or valid math that is validly interpreted. (Listed in increasing order of difficulty.) Stripping out the theories and giving the mere advice alone wouldn’t have nearly the same impact or even the same message; and oddly enough, translating experiments and math into practical advice seems to be a rare niche activity relative to academia. If there’s a distinctive LW style, this is it.
332. The Sin of Underconfidence—When subjects know about a bias or are warned about a bias, overcorrection is not unheard of as an experimental result. That’s what makes a lot of cognitive subtasks so troublesome—you know you’re biased but you’re not sure how much, and if you keep tweaking you may overcorrect. The danger of underconfidence (overcorrecting for overconfidence) is that you pass up opportunities on which you could have been successful; not challenging difficult enough problems; losing forward momentum and adopting defensive postures; refusing to put the hypothesis of your inability to the test; losing enough hope of triumph to try hard enough to win. You should ask yourself “Does this way of thinking make me stronger, or weaker?”
333. Go Forth and Create the Art! - I’ve developed primarily the art of epistemic rationality, in particular, the arts required for advanced cognitive reductionism… arts like distinguishing fake explanations from real ones and avoiding affective death spirals. There is much else that needs developing to create a craft of rationality—fighting akrasia; coordinating groups; teaching, training, verification, and becoming a proper experimental science; developing better introductory literature… And yet it seems to me that there is a beginning barrier to surpass before you can start creating high-quality craft of rationality, having to do with virtually everyone who tries to think lofty thoughts going instantly astray, or indeed even realizing that a craft of rationality exists and that you ought to be studying cognitive science literature to create it. It’s my hope that my writings, as partial as they are, will serve to surpass this initial barrier. The rest I leave to you.
This has been a collection of notes on the assigned sequence for this fortnight. The most important part of the reading group though is discussion, which is in the comments section. Please remember that this group contains a variety of levels of expertise: if a line of discussion seems too basic or too incomprehensible, look around for one that suits you better!
This is the end, beautiful friend!
And because we are taught that imagining “vastly more” is naive. Wise people know that we live in the best possible universe because if something could be improved, someone else would have already improved it. (Who are you to believe that you can improve something that people higher-status than you didn’t?)
This is a “reversed stupidity” against people who believe that vast improvement are trivial. (Usually because they completely ignore coordination problems, perverse incentives, or even the obvious negative side-effects of their improvements. “Why don’t we just abolish money and kill/reeducate all the bad people? Then the world would be a nice place without scarcity.”) But the opposite of “trivial” isn’t “impossible”, and great improvements happen regularly, for example the computers and internet.
Then probably the next answer is that improvement is possible, but only by institutions. Which ignores the fact that (1) the institutions don’t grow on trees, they are created by people, and (2) even the institutions can benefit from having competent people at the right place, so that fact that we need institutions is not an argument against personal competence.
So, how far did we get? I think we don’t even have the “rationality dojos” competing against each other, and I’d say it’s time we made some. Maybe local meetups could be considered such. So, how specifically can we make them “fight” against each other? What would be the tournament about? Maybe something general like “you have two months to do an impressive thing, then report about it, and the jury will choose the most impressive winner”.
Anyone willing to countersignal, and make a well-cooperating rationalist group just to spite the others? ;)
This makes me realize in near mode how long way is there yet ahead of us. Keeping this website interesting and friendly is already a difficult problem; and it’s incomparable with running a country and facing a war. But maybe that’s a problem specific for online environments, and a real country, or a town, or a village, or even a large house would have a different dynamic, because people could meet in person and cooperate at real-life projects (which could teach them the cooperation skills, and select competent people by things other than posturing).
By the way, what is the largest known group of aspiring rationalists living together?
This means that for practical purposes only knowledge of cognitive biases in general is not enough. One needs to have knowledge in domains where they want to achieve something. Using the division of labor, someone can research a topic for the others, and then tell them only as much of the theory as they need to deeply understand the advice.
Which means that a website (or other communication tool) of a rationalist community that actually does something would contain articles on things beyond cognitive biases and math. If you want to build a house for rationalists to live together, you need to research something about house-building. You can delegate the whole decision to one person, if you trust their expertise enough, but if you want to have a community discussion, you need to educate the community about the basics, so they can participate in the debate meaningfully. (Perhaps the lesson should not be called “rational house-building”, but as long as the rationalists want to debate house-building, some lesson is required. Of course, lessons produced outside of the community can be used, no need to reinvent the wheel, unless you believe you can communicate the same insights more efficiently.)
And when many people do this, underconfidence becomes a standard signal of wisdom. (“No, I can’t do anything, because I am not 100% sure it’s the right action, and I am too clever to do a possibly wrong thing.” “Great thinking! Anyone who does anything is clearly not smart enough to realize this.”) Perverse incentives: in many situations, failure is obvious and humiliating, while not taking action is considered normal. It is good to take a step away and realize that in long term the failures are likely to be forgotten, but the successes can become stepping stones to further victories. Even if the failures are more frequent, as long as they are not too costly.
Thank you a lot for providing all these summaries even if they led to little discussion. I valued them a lot!
That’s good enough for me.
Just read Collective Apathy and the Internet and Yvain’s comment about examples when it (‘the internet’) work in coordinating group action. Apparently sometimes it does and it may depend on a few things like critical mass, attention span, identity, emotionality.
Let’s try a poll.
I think that the internet works starts to work when...
...critical mass of community is [pollid:1142]
...effect is visible [pollid:1143]
How relevant do you think are the following effects for the success
Interesting/new [pollid:1144]
Outrageous [pollid:1145]
Differing from standard norms [pollid:1146]
Sad [pollid:1147]
Very beneficial for larger cause [pollid:1148]
What are your ideas? When does the internet work at coordination and when doesn’t it?
“Within a minute” is a bit off; it needs to be long enough that somebody has to keep coming back to see, but short enough that they don’t get bored waiting. More than a minute, less than ten.