Philosopher Dan Williams[1]’s recent post titled “Socialism, self-deception, and spontaneous order” contains some relevant commentary[2]:
Humans are intensely cooperative. Why? What motivates individuals to help and assist others? Why are people often so friendly, generous, and fair-minded?
Genuine altruism plays an important role among close family members. For well-known evolutionary reasons, organisms can propagate their genes by helping relatives who share copies of those genes. This is why the most intense cooperation typically occurs among relatives, and extended kinship networks are such a common form of social organisation.
However, most human cooperation is not altruistic; it is mutualistic. Outside the family, people cooperate for mutual benefit—to achieve fitness-relevant goals they could not achieve independently.
For example, we cooperate for reasons of interdependence. In many contexts, we depend on—that is, have a self-interested stake in the survival and success of—others, so we benefit ourselves by helping them. Although the concept of interdependence makes many people feel warm and fuzzy, it has a harsh logic: if we do not depend on others—e.g., because they are a rival, outsider, or burden—motives to help them often disappear.
We also cooperate so that others might return the favour. This includes long-term reciprocal relationships between spouses and friends, as well as community-wide (“indirect”) reciprocity whereby people aid others so that someone (not necessarily the specific person they helped) will return the favour. Such community-wide cooperation is typically scaffolded by social norms and gossip, which emerge because people personally benefit from designing, enforcing, and conforming to norms and from spreading and consuming juicy tittle-tattle about people’s behaviours and misdeeds.
Relatedly, we cooperate to improve our reputations and win esteem. People benefit from being seen as virtuous, norm-abiding community members. A good reputation is socially rewarded, protects people against collective punishment, and makes them more attractive mates, friends, and allies. Given this, self-sacrifice, benevolence, and generosity are often reputationally lucrative, which explains why—paradoxically—people sometimes compete to be more cooperative than others.
A striking feature of research on the evolution of human cooperation is that the explanations it identifies—for example, mutual benefit, interdependence, reciprocity, reputation, and prestige—often bear little resemblance to people’s own understanding of their cooperative behaviour.
I am not making the trivial point that people are oblivious to the evolutionary reasons for their actions. Even if people’s prosocial instincts are ultimately rooted in the fitness benefits of cooperation, such Darwinian rationales are always distinct from people’s proximate motives. For example, people typically have sex not to reproduce but because sex feels good, even though the Darwinian reason sex feels good is to encourage reproduction. Likewise, people care for their children not to propagate their genes but because they love their children, even though the Darwinian reason for parental love is that it helps people propagate their genes.
Mutualistic cooperation is different: it is not just the Darwinian rationale for cooperation that clashes with our self-understanding but the proximate motives driving our behaviour. Because mutualistic cooperation functions to promote individual self-interest, people are generally motivated to cooperate only when doing so is personally advantageous. That is, unlike sex or the well-being of their children, people do not place a non-instrumental value on mutualistic cooperation; it must be incentivised.
[...]
Nevertheless, when people describetheir cooperative behaviour, they almost neveracknowledge this situation. They depict self-interest as irrelevant to their friendliness, fair-mindedness, and generosity. The help others because others need help, because they care about the “common good”, because it is “the right thing to do”, and so on. They value cooperation not because it constitutes a personally advantageous strategy; they value it as an end in itself.
Once again, the point is not that this self-conception is a lie. People sincerely—indeed, passionately—believe such stories. Moreover, we have evolved to cultivate prosocial traits and dispositions that make such stories plausible. Because it is costly to appear like calculating and selective cooperators, we often cultivate robust prosocial instincts designed to appear uncalculating and unselective. Nevertheless, just like press secretaries are highly skilled at painting their client’s behaviour in the most attractive light, the stories we tell—and believe—about our motives for cooperation are designed not for accuracy but to paint our behaviour in ways that make us look good.
I don’t think “self-deception” is a satisfying answer to why this happens, as if to claim that you just need to realize that you’re secretly causal decision theory inside. It seems to me that this does demonstrate a mismatch, and failing to notice the mismatch is an error, but people who want that better world need not give up on it just because there’s a mismatch. I even agree that things are often optimized to make people look good. But I don’t think it’s correct to jump to “and therefore, people cannot objectively care about each other in ways that are not advantageous to their own personal fitness”. I think there’s a failure of communication, where the perspective he criticizes is broken according to its own values, and part of how it’s broken involves self-deception, but saying that and calling it a day misses most of the interesting patterns in why someone who wants a better world feels drawn to the ideas involved and feels the current organizational designs are importantly broken.
I feel similarly about OP. Like, agree maybe it’s insurance—but, are you sure we’re using the decision theory we want to be here?
another quote from the article you linked:
To be clear, the point is not that people are Machiavellian psychopaths underneath the confabulations and self-narratives they develop. Humans have prosocial instincts, empathy, and an intuitive sense of fairness. The point is rather that these likeable features are inevitably limited, and self-serving motives—for prestige, power, and resources—often play a bigger role in our behaviour than we are eager to admit.
...or approve of? this seems more like a failure to implement ones’ own values! I feel more like the “real me” is the one who Actually Cooperates Because I Care, and the present day me who fails at that does so because of failing to be sufficiently self-and-other-interpretable to be able to demand I do it reliably (but like, this is from a sort of FDT-ish perspective, where when we consider changing this, we’re considering changing all people who would have a similar-to-me thought about this at once to be slightly less discooperative-in-fact). Getting to a point where we can have a better OSGT moral equilibrium (in the world where things weren’t about to go really crazy from AI) would have to be an incremental deescalation of inner vs outer behavior mismatch, but I feel like we ought to be able to move that way in principle, and it seems to me that I endorse the side of this mismatch that this article calls self-deceptive. Yeah, it’s hard to care about everyone, and when the only thing that gives heavy training pressure to do so is an adversarial evaluation game, it’s pretty easy to be misaligned. But I think that’s bad actually, and smoothly, non-abruptly moving to an evaluation environment where matching internal vs external is possible seems like in the non-AI world it would sure be pretty nice!
(edit: at very least in the humans-only scenario, I claim much of the hard part of that is doing this more-transparency-and-prosociality-demanding-environemnt in a way that doesn’t cause a bunch of negative spurious demands, and/or/via just moving the discooperativeness to the choice of what demands become popular. I claim that people currently taking issue with attempts at using increased pressure to create this equilibrium are often noticing ways the more-prosociality-demanding-memes didn’t sufficiently self-reflect to avoid making what are actually in some way just bad demands by more-prosocial-memes’ own standards.)
maybe even in the AI world; it just like, might take a lot longer to do this for humans than we have time for. but maybe it’s needed to solve the problem, idk. getting into the more speculative parts of the point I wanna make here.
Philosopher Dan Williams[1]’s recent post titled “Socialism, self-deception, and spontaneous order” contains some relevant commentary[2]:
Who, somewhat unrelatedly, is a fellow of the LCFI
And is worthwhile to read anyway.
[edit: pinned to profile]
I don’t think “self-deception” is a satisfying answer to why this happens, as if to claim that you just need to realize that you’re secretly causal decision theory inside. It seems to me that this does demonstrate a mismatch, and failing to notice the mismatch is an error, but people who want that better world need not give up on it just because there’s a mismatch. I even agree that things are often optimized to make people look good. But I don’t think it’s correct to jump to “and therefore, people cannot objectively care about each other in ways that are not advantageous to their own personal fitness”. I think there’s a failure of communication, where the perspective he criticizes is broken according to its own values, and part of how it’s broken involves self-deception, but saying that and calling it a day misses most of the interesting patterns in why someone who wants a better world feels drawn to the ideas involved and feels the current organizational designs are importantly broken.
I feel similarly about OP. Like, agree maybe it’s insurance—but, are you sure we’re using the decision theory we want to be here?
another quote from the article you linked:
...or approve of? this seems more like a failure to implement ones’ own values! I feel more like the “real me” is the one who Actually Cooperates Because I Care, and the present day me who fails at that does so because of failing to be sufficiently self-and-other-interpretable to be able to demand I do it reliably (but like, this is from a sort of FDT-ish perspective, where when we consider changing this, we’re considering changing all people who would have a similar-to-me thought about this at once to be slightly less discooperative-in-fact). Getting to a point where we can have a better OSGT moral equilibrium (in the world where things weren’t about to go really crazy from AI) would have to be an incremental deescalation of inner vs outer behavior mismatch, but I feel like we ought to be able to move that way in principle, and it seems to me that I endorse the side of this mismatch that this article calls self-deceptive. Yeah, it’s hard to care about everyone, and when the only thing that gives heavy training pressure to do so is an adversarial evaluation game, it’s pretty easy to be misaligned. But I think that’s bad actually, and smoothly, non-abruptly moving to an evaluation environment where matching internal vs external is possible seems like in the non-AI world it would sure be pretty nice!
(edit: at very least in the humans-only scenario, I claim much of the hard part of that is doing this more-transparency-and-prosociality-demanding-environemnt in a way that doesn’t cause a bunch of negative spurious demands, and/or/via just moving the discooperativeness to the choice of what demands become popular. I claim that people currently taking issue with attempts at using increased pressure to create this equilibrium are often noticing ways the more-prosociality-demanding-memes didn’t sufficiently self-reflect to avoid making what are actually in some way just bad demands by more-prosocial-memes’ own standards.)
maybe even in the AI world; it just like, might take a lot longer to do this for humans than we have time for. but maybe it’s needed to solve the problem, idk. getting into the more speculative parts of the point I wanna make here.