[SEQ RERUN] Ends Don’t Justify Means (Among Humans)
Today’s post, Ends Don’t Justify Means (Among Humans) was originally published on 14 October 2008. A summary (taken from the LW wiki):
There may be consequentialist reasons for why humans should not reason in an explicitly consequentialist way. We are running on corrupted hardware, and it may seem to us that something benefits others when it does not.
Discuss the post here (rather than in the comments to the original post).
This post is part of the Rerunning the Sequences series, where we’ll be going through Eliezer Yudkowsky’s old posts in order so that people who are interested can (re-)read and discuss them. The previous post was Why Does Power Corrupt?, and you can use the sequence_reruns tag or rss feed to follow the rest of the series.
Sequence reruns are a community-driven effort. You can participate by re-reading the sequence post, discussing it here, posting the next day’s sequence reruns post, or summarizing forthcoming articles on the wiki. Go here for more details, or to have meta discussions about the Rerunning the Sequences series.
The tribe which has the rule “For the good of the tribe, do not cheat to seize power even when it would provide a net benefit to the tribe.” performs better than the tribe which has the rule “Cheat to seize power when it is in the benefit of the tribe.”
One reason for this is that if a significant proportion of the tribe agrees on what the best course of action is, no usurper is needed- and if only a tiny fraction can agree on what is best, the remainder cannot tell if they are acting in good faith or not. A tribe which trusts everyone who says ‘trust me’ is betrayed much more than a tribe which trusts everyone in a manner consistent what the evidence suggests their trustworthiness is.
Of course, cheating is already breaking the rules!
Right- but in some cases breaking the rules is sanctioned by the rules.