(Moral) Truth in Fiction?

Eliezer Yudkowsky9 Feb 2009 17:26 UTC
25 points
82 comments7 min readLW linkArchive

A comment by Anonymous on Three Worlds Collide:

After reading this story I feel myself agreeing with Eliezer more on his views and that seems to be a sign of manipulation and not of rationality.

Philosophy expressed in form of fiction seems to have a very strong effect on people—even if the fiction isn’t very good (ref. Ayn Rand).

Robin has similar qualms:

Since people are inconsistent but reluctant to admit that fact, their moral beliefs can be influenced by which moral dilemmas they consider in what order, especially when written by a good writer. I expect Eliezer chose his dilemmas in order to move readers toward his preferred moral beliefs, but why should I expect those are better moral beliefs than those of all the other authors of fictional moral dilemmas?

If I’m going to read a literature that might influence my moral beliefs, I’d rather read professional philosophers and other academics making more explicit arguments.

I replied that I had taken considerable pains to set out the explicit arguments before daring to publish the story. And moreover, I had gone to considerable length to present the Superhappy argument in the best possible light. (The opposing viewpoint is the counterpart of the villain; you want it to look as reasonable as possible for purposes of dramatic conflict, the same principle whereby Frodo confronts the Dark Lord Sauron rather than a cockroach.)

Robin didn’t find this convincing:

I don’t think readers should much let down their guard against communication modes where sneaky persuasion is more feasible simply because the author has made some more explicit arguments elsewhere… Academic philosophy offers exemplary formats and styles for low-sneak ways to argue about values.

I think that this understates the power and utility of fiction. I once read a book that was called something like “How to Read” (no, not “How to Read a Book”) which said that nonfiction was about communicating knowledge, while fiction was about communicating experience.

If I want to communicate something about the experience of being a rationalist, I can best do it by writing a short story with a rationalist character. Not only would identical abstract statements about proper responses have less impact, they wouldn’t even communicate the same thought.

From The Failures of Eld Science:

″...Work expands to fill the time allotted, as the saying goes. But people can think important thoughts in far less than thirty years, if they expect speed of themselves.” Jeffreyssai suddenly slammed down a hand on the arm of Brennan’s chair. “How long do you have to dodge a thrown knife?”

“Very little time, sensei!”

“Less than a second! Two opponents are attacking you! How long do you have to guess who’s more dangerous?”

“Less than a second, sensei!”

“The two opponents have split up and are attacking two of your girlfriends! How long do you have to decide which one you truly love?”

“Less than a second, sensei!”

“A new argument shows your precious theory is flawed! How long does it take you to change your mind?”

“Less than a second, sensei!”

“WRONG! DON’T GIVE ME THE WRONG ANSWER JUST BECAUSE IT FITS A CONVENIENT PATTERN AND I SEEM TO EXPECT IT OF YOU! How long does it really take, Brennan?”

Sweat was forming on Brennan’s back, but he stopped and actually thought about it -

“ANSWER, BRENNAN!”

“No sensei! I’m not finished thinking sensei! An answer would be premature! Sensei!”

“Very good! Continue! But don’t take thirty years!”

This is an experience about how to avoid completing the pattern when the pattern happens to be blatantly wrong, and how to think quickly without thinking too quickly.

Forget the question of whether you can write the equivalent abstract argument that communicates the same thought in less space. Can you do it at all? Is there any series of abstract arguments that creates the same learning experience in the reader? Entering a series of believed propositions into your belief pool is not the same as feeling yourself in someone else’s shoes, and reacting to the experience, and forming an experiential skill-memory of how to do it next time.

And it seems to me that to communicate experience is a valid form of moral argument as well.

Uncle Tom’s Cabin was not just a historically powerful argument against slavery, it was a valid argument against slavery. If human beings were constructed without mirror neurons, if we didn’t hurt when we see a nonenemy hurting, then we would exist in the reference frame of a different morality, and we would decide what to do by asking a different question, “What should* we do?” Without that ability to sympathize, we might think that it was perfectly all right* to keep slaves. (See Inseparably Right and No License To Be Human.)

Putting someone into the shoes of a slave and letting their mirror neurons feel the suffering of a husband separated from a wife, a mother separated from a child, a man whipped for refusing to whip a fellow slave—it’s not just persuasive, it’s valid. It fires the mirror neurons that physically implement that part of our moral frame.

I’m sure many have turned against slavery without reading Uncle Tom’s Cabin—maybe even due to purely abstract arguments, without ever seeing the carving “Am I Not a Man and a Brother?” But for some people, or for a not-much-different intelligent species, reading Uncle Tom’s Cabin might be the only argument that can turn you against slavery. Any amount of abstract argument that didn’t fire the experiential mirror neurons, would not activate the part of your implicit should-function that disliked slavery. You would just seem to be making a good profit on something you owned.

Can fiction be abused? Of course. Suppose that blacks had no subjective experiences. Then Uncle Tom’s Cabin would have been a lie in a deeper sense than being fictional, and anyone moved by it would have been deceived.

Or to give a more subtle case not involving a direct “lie” of this sort: On the SL4 mailing list, Stuart Armstrong posted an argument against TORTURE in the infamous Torture vs. Dust Specks debate, consisting of a short story describing the fate of the person to be tortured. My reply was that the appropriate counterargument would be 3^^^3 stories about someone getting a dust speck in their eye. I actually did try to send a long message consisting only of

DUST SPECK
DUST SPECK
DUST SPECK
DUST SPECK
DUST SPECK
DUST SPECK

for a thousand lines or so, but the mailing software stopped it. (Ideally, I should have created a webpage using Javascript and bignums, that, if run on a sufficiently large computer, would print out exactly 3^^^3 copies of a brief story about someone getting a dust speck in their eye. It probably would have been the world’s longest finite webpage. Alas, I lack time for many of my good ideas.)

Then there’s the sort of standard polemic used in e.g. Atlas Shrugged (as well as many less famous pieces of science fiction) in which Your Beliefs are put into the minds of strong empowered noble heroes, and the Opposing Beliefs are put into the mouths of evil and contemptible villains, and then the consequences of Their Way are depicted as uniformly disastrous while Your Way offers butterflies and apple pie. That’s not even subtle, but it works on people predisposed to hear the message.

But to entirely turn your back on fiction is, I think, taking it too far. Abstract argument can be abused too. In fact, I would say that abstract argument is if anything easier to abuse because it has more degrees of freedom. Which is easier, to say “Slavery is good for the slave”, or to write a believable story about slavery benefiting the slave? You can do both, but the second is at least more difficult; your brain is more likely to notice the non-sequiturs when they’re played out as a written experience.

Stories may not get us completely into Near mode, but they get us closer into Near mode than abstract argument. If it’s words on paper, you can end up believing that you ought to do just about anything. If you’re in the shoes of a character encountering the experience, your reactions may be harder to twist.

Contrast a verbal argument against the verbal belief that “non-Catholics go to Hell”; versus reading a story about a good and decent person, who happens to be a Protestant, and dies trying to save a child’s life, who is condemned to hell and has molten lead poured down her throat; versus the South Park episode where a crowd of newly dead souls is at the entrance to hell, and the Devil says, “Sorry, it was the Mormons” and everyone goes “Awwwww...”

Yes, abstraction done right can keep you going where concrete visualization breaks down—the torture vs. dust specks thing being an archetypal example; you can’t actually visualize that many dust specks, but if you try to choose SPECKS you’ll end up with circular preferences. But so far as I can organize my metaethics, the ground level of morality lies in our preferences over particular, concrete situations—and when these can be comprehended as concrete images at all, it’s best to visualize them as concretely as possible. Unless we know specifically where the concrete image is going wrong, and have to apply an abstract correction. The moral abstraction is built on top of the ground level.

I am also, of course, worried about the idea that stories aren’t “respectable” because they don’t look sufficiently solemn and dull; or the idea that something isn’t “respectable” if can be understood by a mere popular audience. Yes, there are technical fields that are genuinely impossible to explain to your grandmother in an hour; but ceteris paribus, people who can write at a more popular level without distorting technical reality are performing a huge service to that field. I’ve heard that Carl Sagan was held in some disrepute by his peers for the crime of speaking to the general public. If true, this is merely stupid.

Explaining things is hard. Explainers need every tool they can get their hands on—as a matter of public interest.

And in moral philosophy—well, I suppose it could be the case that moral philosophers have discovered moral truths that are deductive consequences of most humans’ moral frames, but which are so difficult and technical that they simply can’t be explained to a popular audience within a one-hour lecture. But it would be a tad more suspicious than the corresponding case in, say, physics.

I realize that I speak as someone who does a lot of popularizing, but even so—fiction ought to be a respectable form of moral argument. And a respectable way of communicating experiences, in particular the experience of applying certain types of thinking skills.

I’ve always been of two minds about publishing longer fiction pieces about the future and its consequences. Not so much because of the potential for abuse, but because even when not abused, fiction can still bypass critical faculties and end up poured directly into the brains of at least some readers. Telling people about the logical fallacy of generalization from fictional evidence doesn’t make it go away; people may just go on generalizing from the story as though they had actually seen it happen. And you simply can’t have a story that’s a rational projection; it’s not just a matter of plot, it’s a matter of the story needing to be specific, rather than depicting a state of epistemic uncertainty.

But to make shorter philosophical points? Sure.

And… oh, what the hell. Just on the off-chance, are there any OB readers who could get a good movie made? Either outside Hollywood, or able to bypass the usual dumbing-down process that creates a money-losing flop? The probabilities are infinitesimal, I know, but I thought I’d check.

Eliezer Yudkowsky9 Feb 2009 17:26 UTC
25 points
82 comments7 min readLW linkArchive