Examples of some common ways that people sometimes find Singularity scenarios disorienting:
When a person loses their childhood religion, there’s often quite a bit of bucket error. A person updates on the true fact “Jehovah is not a good explanation of the fossil record” and accidentally confuses that true fact with any number of other things, such as “and so I’m not allowed to take my friends’ lives and choices as real and meaningful.”
I claimed above that “coming to take singularity scenarios seriously” seems in my experience to often cause even more disruption / bucket errors / confusions / false beliefs than does “losing a deeply held childhood religion.” I’d like to elaborate on that here by listing some examples of the kinds of confusions/errors I often encounter.
None of these are present in everyone who encounters Singularity scenarios, or even in most people who encounter it. Still, each confusion below is one where I’ve seen it or near-variants of it multiple times.
Also note that all of these things are “confusions”, IMO. People semi-frequently have them at the beginning and then get over them. These are not the POV I would recommend or consider correct—more like the opposite—and I personally think each stems from some sort of fixable thinking error.)
The imagined stakes in a singularity are huge. Common confusions related to this:
Confusion about whether it is okay to sometimes spend money/time/etc. on oneself, vs. having to give it all to attempting to impact the future.
Confusion about whether one wants to take in singularity scenarios, given that then maybe one will “have to” (move across the country / switch jobs / work all the time / etc.)
Confusion about whether it is still correct to follow common sense moral heuristics, given the stakes.
Confusion about how to enter “hanging out” mode, given the stakes and one’s panic. (“Okay, here I am at the beach with my friends, like my todo list told me to do to avoid burnout. But how is it that I used to enjoy their company? They seem to be making meaningless mouth-noises that have nothing to do with the thing that matters…”)
Confusion about how to take an actual normal interest in one’s friends’ lives, or one’s partner’s lives, or one’s Lyft drivers’ lives, or whatever, given that within the person’s new frame, the problems they are caught up in seem “small” or “irrelevant” or to have “nothing to do with what matters”.
The degrees of freedom in “what should a singularity maybe do with the future?” are huge. And people are often morally disoriented by that part.
Should we tile the universe with a single repeated mouse orgasm, or what?
Are we allowed to want humans and ourselves and our friends to stay alive?
Is there anything we actually want? Or is suffering bad without anything being better-than-nothing?
If I can’t concretely picture what I’d do with a whole light-cone (maybe because it is vastly larger than any time/money/resources I’ve ever personally obtained feedback from playing with) -- should I feel that the whole future is maybe meaningless and no good?
The world a person finds themselves in once they start taking Singularity scenarios seriously is often quite different from what the neighbors think, which itself can make things hard
Can I have a “real” conversation with my friends? Should I feel crazy? Should I avoid taking all this in on a visceral level so that I’ll stay mentally in the same world as my friends?
How do I keep regarding other peoples’ actions as good and reasonable?
The imagined scales are very large, with the result that one can less assume “things are locally this way” is an adequate model.
Given this, should I get lost in “what about simulations / anthropics” to the point of becoming confused about normal day-today events?
In order to imagine this stuff, folks need to take seriously reasoning that is neither formal mathematics, nor vetted by the neighbors or academia, nor strongly based in empirical feedback loops.
Given this, shall I go ahead and take random piles of woo seriously also?
There are lots more where these came from, but I’m hoping this gives some flavor, and makes it somewhat plausible why I’m claiming that “coming to take singularity scenarios seriously can be pretty disruptive to common sense,” and such that it might be nice to try having a “bridge” that can help people lose less of the true parts of common sense as their world changes (much as it might be nice for someone who has just lost their childhood religion to have a bridge to “okay, here are some other atheists, and they don’t think that God is why they should get up in the morning and care about others, but they do still seem to think they should get up in the morning and care about others”).
and makes it somewhat plausible why I’m claiming that “coming to take singularity scenarios seriously can be pretty disruptive to common sense,” and such that it might be nice to try having a “bridge” that can help people lose less of the true parts of common sense as their world changes
Can you say a bit more about how CFAR helps people do this? Some of the “confusions” you mentioned are still confusing to me. Are they no longer confusing to you? If so, can you explain how that happened and what you ended up thinking on each of those topics? For example lately I’m puzzling over something related to this:
Given this, should I get lost in “what about simulations / anthropics” to the point of becoming confused about normal day-today events?
[Possibly digging a bit too far into the specifics so no worries if you’d rather bow out.]
Do you think these confusions[1] are fairly evenly dispersed throughout the community (besides what you already mentioned: “People semi-frequently have them at the beginning and then get over them.”)?
Two casual observations: (A) the confusions seem less common among people working full-time at EA/Rationalist/x-risk/longtermist organisation than in other people who “take singularity scenarios seriously.”[2] (B) I’m very uncertain but they also seem less prevalent to me in the EA community than the rationalist community (to the extent the communities can be separated).[3] [4]
Do A and B sound right to you? If so, do you have a take on why that is?
If A or B *are* true, do you think this is in any part caused by the relative groups taking the singularity [/x-risk/the future/the stakes] less seriously? If so, are there important costs from this?
[1] Using your word while withholding my own judgment as to whether every one of these is actually a confusion.
[2] If you’re right that a lot of people have them at the beginning and then get over them, a simple potential explanation would be that by the time you’re working at one of these orgs, that’s already happened.
Other hypothesis: (a) selection effects; (b) working FT in the community gives you additional social supports and makes it more likely others will notice if you start spiraling; (c) the cognitive dissonance with the rest of society is a lot of what’s doing the damage. It’s easier to handle this stuff psychologically if the coworkers you see every day also take the singularity seriously.[i]
[3] For example perhaps less common at Open Phil, GPI, 80k, and CEA than CFAR and MIRI but I also think this holds outside of professional organisations.
[4] One potential reason for this is that a lot of EA ideas are more “in the air” than rationalist/singularity ones. So a lot of EAs may have had their ‘crisis of faith’ before arriving in the community. (For example, I know plenty of EAs (myself included) who did some damage to themselves in their teens or early twenties by “taking Peter Singer really seriously.”
[i] I’ve seen this kind of dissidence offered as a (partial) explanation of why PTSD has become so common among veterans & why it’s so hard for them to reintegrate after serving a combat tour. No clue if the source is reliable/widely held/true. It’s been years but I think I got it from Odysseus in America or perhaps its predecessor, Achilles in Vietnam.
This seemed really useful. I suspect you’re planning to write up something like this at some point down the line but wanted to suggest posting this somewhere more prominent in the meantime (otoh, idea inoculation, etc.)
The state of confusion you’re describing sounds a lot like Kegan’s 4.5 nihilism (pretty much everything at meaningness.com is relevant). A person’s values have been demolished by a persuasive argument, but they haven’t yet internalized that people are “allowed” to create their own systems and values. Alright.
1. I assume that LW-adjacent people should actually be better at guiding people out of this stage, because a lot of people in the community have gone through the same process and there is an extensive body of work on the topic (Eliezer’s sequences on human values, David Chapman’s work, Scott Alexander’s posts on effective altruism / axiology-vs-morality / etc).
2. I also assume that in general we want people to go through this process – it is a necessary stage of adult development.
Given this, I’m leaning towards “guiding people towards nihilism is good as long as you don’t leave them in the philosophical dark re/ how to get out of it”. So, taking a random smart person, persuading them they should care about Singularity, and leaving – this isn’t great. But introducing people to AI risk in the context of LW seems much more benign to me.
Examples of some common ways that people sometimes find Singularity scenarios disorienting:
When a person loses their childhood religion, there’s often quite a bit of bucket error. A person updates on the true fact “Jehovah is not a good explanation of the fossil record” and accidentally confuses that true fact with any number of other things, such as “and so I’m not allowed to take my friends’ lives and choices as real and meaningful.”
I claimed above that “coming to take singularity scenarios seriously” seems in my experience to often cause even more disruption / bucket errors / confusions / false beliefs than does “losing a deeply held childhood religion.” I’d like to elaborate on that here by listing some examples of the kinds of confusions/errors I often encounter.
None of these are present in everyone who encounters Singularity scenarios, or even in most people who encounter it. Still, each confusion below is one where I’ve seen it or near-variants of it multiple times.
Also note that all of these things are “confusions”, IMO. People semi-frequently have them at the beginning and then get over them. These are not the POV I would recommend or consider correct—more like the opposite—and I personally think each stems from some sort of fixable thinking error.)
The imagined stakes in a singularity are huge. Common confusions related to this:
Confusion about whether it is okay to sometimes spend money/time/etc. on oneself, vs. having to give it all to attempting to impact the future.
Confusion about whether one wants to take in singularity scenarios, given that then maybe one will “have to” (move across the country / switch jobs / work all the time / etc.)
Confusion about whether it is still correct to follow common sense moral heuristics, given the stakes.
Confusion about how to enter “hanging out” mode, given the stakes and one’s panic. (“Okay, here I am at the beach with my friends, like my todo list told me to do to avoid burnout. But how is it that I used to enjoy their company? They seem to be making meaningless mouth-noises that have nothing to do with the thing that matters…”)
Confusion about how to take an actual normal interest in one’s friends’ lives, or one’s partner’s lives, or one’s Lyft drivers’ lives, or whatever, given that within the person’s new frame, the problems they are caught up in seem “small” or “irrelevant” or to have “nothing to do with what matters”.
The degrees of freedom in “what should a singularity maybe do with the future?” are huge. And people are often morally disoriented by that part. Should we tile the universe with a single repeated mouse orgasm, or what?
Are we allowed to want humans and ourselves and our friends to stay alive? Is there anything we actually want? Or is suffering bad without anything being better-than-nothing?
If I can’t concretely picture what I’d do with a whole light-cone (maybe because it is vastly larger than any time/money/resources I’ve ever personally obtained feedback from playing with) -- should I feel that the whole future is maybe meaningless and no good?
The world a person finds themselves in once they start taking Singularity scenarios seriously is often quite different from what the neighbors think, which itself can make things hard
Can I have a “real” conversation with my friends? Should I feel crazy? Should I avoid taking all this in on a visceral level so that I’ll stay mentally in the same world as my friends?
How do I keep regarding other peoples’ actions as good and reasonable? The imagined scales are very large, with the result that one can less assume “things are locally this way” is an adequate model.
Given this, should I get lost in “what about simulations / anthropics” to the point of becoming confused about normal day-today events?
In order to imagine this stuff, folks need to take seriously reasoning that is neither formal mathematics, nor vetted by the neighbors or academia, nor strongly based in empirical feedback loops.
Given this, shall I go ahead and take random piles of woo seriously also?
There are lots more where these came from, but I’m hoping this gives some flavor, and makes it somewhat plausible why I’m claiming that “coming to take singularity scenarios seriously can be pretty disruptive to common sense,” and such that it might be nice to try having a “bridge” that can help people lose less of the true parts of common sense as their world changes (much as it might be nice for someone who has just lost their childhood religion to have a bridge to “okay, here are some other atheists, and they don’t think that God is why they should get up in the morning and care about others, but they do still seem to think they should get up in the morning and care about others”).
Can you say a bit more about how CFAR helps people do this? Some of the “confusions” you mentioned are still confusing to me. Are they no longer confusing to you? If so, can you explain how that happened and what you ended up thinking on each of those topics? For example lately I’m puzzling over something related to this:
[Possibly digging a bit too far into the specifics so no worries if you’d rather bow out.]
Do you think these confusions[1] are fairly evenly dispersed throughout the community (besides what you already mentioned: “People semi-frequently have them at the beginning and then get over them.”)?
Two casual observations: (A) the confusions seem less common among people working full-time at EA/Rationalist/x-risk/longtermist organisation than in other people who “take singularity scenarios seriously.”[2] (B) I’m very uncertain but they also seem less prevalent to me in the EA community than the rationalist community (to the extent the communities can be separated).[3] [4]
Do A and B sound right to you? If so, do you have a take on why that is?
If A or B *are* true, do you think this is in any part caused by the relative groups taking the singularity [/x-risk/the future/the stakes] less seriously? If so, are there important costs from this?
[1] Using your word while withholding my own judgment as to whether every one of these is actually a confusion.
[2] If you’re right that a lot of people have them at the beginning and then get over them, a simple potential explanation would be that by the time you’re working at one of these orgs, that’s already happened.
Other hypothesis: (a) selection effects; (b) working FT in the community gives you additional social supports and makes it more likely others will notice if you start spiraling; (c) the cognitive dissonance with the rest of society is a lot of what’s doing the damage. It’s easier to handle this stuff psychologically if the coworkers you see every day also take the singularity seriously.[i]
[3] For example perhaps less common at Open Phil, GPI, 80k, and CEA than CFAR and MIRI but I also think this holds outside of professional organisations.
[4] One potential reason for this is that a lot of EA ideas are more “in the air” than rationalist/singularity ones. So a lot of EAs may have had their ‘crisis of faith’ before arriving in the community. (For example, I know plenty of EAs (myself included) who did some damage to themselves in their teens or early twenties by “taking Peter Singer really seriously.”
[i] I’ve seen this kind of dissidence offered as a (partial) explanation of why PTSD has become so common among veterans & why it’s so hard for them to reintegrate after serving a combat tour. No clue if the source is reliable/widely held/true. It’s been years but I think I got it from Odysseus in America or perhaps its predecessor, Achilles in Vietnam.
This seemed really useful. I suspect you’re planning to write up something like this at some point down the line but wanted to suggest posting this somewhere more prominent in the meantime (otoh, idea inoculation, etc.)
The state of confusion you’re describing sounds a lot like Kegan’s 4.5 nihilism (pretty much everything at meaningness.com is relevant). A person’s values have been demolished by a persuasive argument, but they haven’t yet internalized that people are “allowed” to create their own systems and values. Alright.
1. I assume that LW-adjacent people should actually be better at guiding people out of this stage, because a lot of people in the community have gone through the same process and there is an extensive body of work on the topic (Eliezer’s sequences on human values, David Chapman’s work, Scott Alexander’s posts on effective altruism / axiology-vs-morality / etc).
2. I also assume that in general we want people to go through this process – it is a necessary stage of adult development.
Given this, I’m leaning towards “guiding people towards nihilism is good as long as you don’t leave them in the philosophical dark re/ how to get out of it”. So, taking a random smart person, persuading them they
should care about Singularity, and leaving – this isn’t great. But introducing people to AI risk in the context of LW seems much more benign to me.