Over the last 12 years, I’ve chatted with small hundreds of people who were somewhere “in process” along the path toward “okay I guess I should take Singularity scenarios seriously.” From watching them, my guess is that the process of coming to take Singularity scenarios seriously is often even more disruptive than is losing a childhood religion. Among many other things, I have seen it sometimes disrupt:
I feel like I was hit by most of these disruptions myself, and eventually managed to overcome them. But the exact nature of how exactly I overcame them, suggests to me that there might be one more piece to the puzzle which hasn’t been mentioned here.
A concept which I’ve seen thrown around in a few places is that of an “exile-driven life”; “exile” referring to the Internal Family Systems notion of strong painful feelings which a person is desperate to keep buried. Your life or some aspect of your life being exile-driven, means that keeping those painful feelings suppressed is one of the primary motivations behind your choices. The alcoholic who drinks to make their feelings of shame go away is exile-driven, but one can also have an exile-driven career that looks successful from the outside, or an exile-driven relationship where someone is primarily in the relationship for the sake of e.g. getting validation from their partner, and gets desperate whenever they don’t get enough of it.
In retrospect, it looks to me like most of my disruptions—such as losing the belief of having a right to rest etc. - were ultimately linked to strong feelings of moral obligation, guilt, and worthlessness which have also popped up in other contexts. For example, it has happened more than once that a friend has gotten very depressed and suicidal, and then clutched onto me for help; and this has triggered exactly the same kind of reasoning as the various Singularity scenarios. “What right do I have to rest when this other person is much more badly off”, and other classic codependency symptoms. (Looking at that list of codependency symptoms actually makes for a very interesting parallel to “Singularity disorder”, now that I think of it.)
Now, I do agree that there’s something to the “eliminating antibodies” framing—in each of those cases, there have been related thoughts about consequentialism and (this was particularly toxic) heroic responsibility saying that yes, if I don’t manage to help this person, then their suffering and possibly death is my fault.
But the “eliminating antibodies” framing is something that suggests that this is something that could happen to anyone. And maybe it could: part of my recovery involved starting to explicitly reject excessive consequentialism and utilitarianism in my thinking. Still, it wasn’t until I found ways to address the underlying emotional flaws themselves, that the kinds of failure modes that you described also started fixing themselves more thoroughly.
So at least my own experience was less of “eliminating these antibodies caused me to overgeneralize factual beliefs”, as “there were pre-existing parts of my mind that believed that I was worthless, and all the rationalist stuff handed them even more evidence that they could use for making that case, eliminating existing defenses against the belief”. If I hadn’t had those pre-existing vulnerabilities, I suspect that I wouldn’t have been disrupted to the same extent.
Qiaochu and others have been making the observation that the rationalist community seems to have a large share of people who are traumatized; it’s been remarked that self-improvement communities in general attract the walking wounded. At my IFS training, it was remarked that manager parts that are struggling to keep exiles in bay tend to be really strongly attracted into any systems which offer a promise of control and predictability, such as what you might get from the original Sequences—“here are the mathematically correct ways of reasoning and acting, just follow these instructions and you’re doing as well as a human can!”. There’s the thought that if only you can work yourself hard enough, and follow the dictates of this new system faithfully enough, then the feelings of guilt and worthlessness will stop. But since consequentialism is more demanding than what any human is ever capable of, you can never say “okay, now I’ve done enough and can rest”, and those feelings of worthlessness will just continue to recur.
This would suggest that not only are there pre-existing vulnerabilities that make some people more susceptible to being disrupted by rationalist memes, those are also exactly the same kinds of people who frequently get drawn to rationalist memes, since in the view of some of their parts, the “disruption” is actually a way to redeem themselves.
I feel like I was hit by most of these disruptions myself, and eventually managed to overcome them. But the exact nature of how exactly I overcame them, suggests to me that there might be one more piece to the puzzle which hasn’t been mentioned here.
A concept which I’ve seen thrown around in a few places is that of an “exile-driven life”; “exile” referring to the Internal Family Systems notion of strong painful feelings which a person is desperate to keep buried. Your life or some aspect of your life being exile-driven, means that keeping those painful feelings suppressed is one of the primary motivations behind your choices. The alcoholic who drinks to make their feelings of shame go away is exile-driven, but one can also have an exile-driven career that looks successful from the outside, or an exile-driven relationship where someone is primarily in the relationship for the sake of e.g. getting validation from their partner, and gets desperate whenever they don’t get enough of it.
In retrospect, it looks to me like most of my disruptions—such as losing the belief of having a right to rest etc. - were ultimately linked to strong feelings of moral obligation, guilt, and worthlessness which have also popped up in other contexts. For example, it has happened more than once that a friend has gotten very depressed and suicidal, and then clutched onto me for help; and this has triggered exactly the same kind of reasoning as the various Singularity scenarios. “What right do I have to rest when this other person is much more badly off”, and other classic codependency symptoms. (Looking at that list of codependency symptoms actually makes for a very interesting parallel to “Singularity disorder”, now that I think of it.)
Now, I do agree that there’s something to the “eliminating antibodies” framing—in each of those cases, there have been related thoughts about consequentialism and (this was particularly toxic) heroic responsibility saying that yes, if I don’t manage to help this person, then their suffering and possibly death is my fault.
But the “eliminating antibodies” framing is something that suggests that this is something that could happen to anyone. And maybe it could: part of my recovery involved starting to explicitly reject excessive consequentialism and utilitarianism in my thinking. Still, it wasn’t until I found ways to address the underlying emotional flaws themselves, that the kinds of failure modes that you described also started fixing themselves more thoroughly.
So at least my own experience was less of “eliminating these antibodies caused me to overgeneralize factual beliefs”, as “there were pre-existing parts of my mind that believed that I was worthless, and all the rationalist stuff handed them even more evidence that they could use for making that case, eliminating existing defenses against the belief”. If I hadn’t had those pre-existing vulnerabilities, I suspect that I wouldn’t have been disrupted to the same extent.
Qiaochu and others have been making the observation that the rationalist community seems to have a large share of people who are traumatized; it’s been remarked that self-improvement communities in general attract the walking wounded. At my IFS training, it was remarked that manager parts that are struggling to keep exiles in bay tend to be really strongly attracted into any systems which offer a promise of control and predictability, such as what you might get from the original Sequences—“here are the mathematically correct ways of reasoning and acting, just follow these instructions and you’re doing as well as a human can!”. There’s the thought that if only you can work yourself hard enough, and follow the dictates of this new system faithfully enough, then the feelings of guilt and worthlessness will stop. But since consequentialism is more demanding than what any human is ever capable of, you can never say “okay, now I’ve done enough and can rest”, and those feelings of worthlessness will just continue to recur.
This would suggest that not only are there pre-existing vulnerabilities that make some people more susceptible to being disrupted by rationalist memes, those are also exactly the same kinds of people who frequently get drawn to rationalist memes, since in the view of some of their parts, the “disruption” is actually a way to redeem themselves.