I think there’s a bunch of useful stuff here. In particular, I think that decisions driven by deep-rooted fear are often very counterproductive, and that many rationalists often have “emergency mobilization systems” running in ways which aren’t conducive to good long-term decision-making. I also think that paying attention to bodily responses is a great tool for helping fix this (and in fact was helpful for me in defusing annoyance when reading this post). But I want to push back on the way in which it’s framed in various places as all-or-nothing: exit the game, or keep playing. Get sober, or stay drunk. Hallucination, not real fear.
In fact, you can do good and important work while also gradually coming to terms with your emotions, trying to get more grounded, and noticing when you’re making decisions driven by visceral fear and taking steps to fix that. Indeed, I expect that almost all good and important work throughout history has been done by people who are at various stages throughout that process, rather than people who first dealt with their traumas and only then turned to the work. (EDIT: in a later comment, Valentine says he doesn’t endorse the claim that people should deal with traumas before doing the work, but does endorse the claim that people should recognize the illusion before doing the work. So better to focus on the latter (I disagree with both).)
(This seems more true for concrete research, and somewhat (but less) true for thinking about high-level strategy. In general it seems that rationalists spend way too much of their time thinking about high-level strategic considerations, and I agree with some of Valentine’s reasoning about why this happens. Instead I’d endorse people trying be much more focused on making progress in a few concrete areas, rather than trying to track everything which they think might be relevant to AI risk. E.g. acceleration is probably bad, but it’s fundamentally a second-order effect, and the energy focused on all but the biggest individual instances of acceleration would probably be better used to focus on first-order effects.)
In other words, I want to offer people the affordance to take on board the (many) useful parts of Valentine’s post without needing to buy into the overall frame in which your current concerns are just a game, and your fear is just a manifestation of trauma.
(Relatedly, from my vantage point it seems that “you need to do the trauma processing first and only then do useful work” is a harmful self-propagating meme in a very similar way as “you need to track and control every variable in order for AI to go well”. Both identify a single dominant consideration which requires your full focus and takes precedence over all others. However, I still think that the former is directionally correct for most rationalists, just as the latter is directionally correct for most non-rationalists.)
it seems that “you need to do the trauma processing first and only then do useful work” is a harmful self-propagating meme in a very similar way as “you need to track and control every variable in order for AI to go well”
This. Trauma processing is just as prone to ouroboros-ing as x-risk work, if not more so.
Wouldn’t it be relevant in that someone could recognize unproductive, toxic dynamics in their concerns about AI risk as per your point (if I understand you correctly), decide to process trauma first and then get stuck in the same sorts of traps? While “I’m traumatized and need to fix it before I can do anything” may not sound as flashy as “My light cone is in danger from unaligned, high-powered AI and I need to fix that before I can do anything”, it’s just as capable of paralyzing a person, and I speak both from my own past mistakes and from those of multiple friends.
If you just go around healing traumas willy-nilly, then you might not ever see through any particular illusion like this one if it’s running in you.
Kind of like, generically working on trauma processing in general might or might not help an alcoholic quit drinking. There’s some reason for hope, but it’s possible to get lost in loops of navel-gazing, especially if they never ever even admit to themselves that they have a problem.
But if it’s targeted, the addiction basically doesn’t stand a chance.
I’m not trying to say “Just work on traumas and be Fully Healed™ before working on AI risk.”
I’m saying something much, much more precise.
I do in fact think there’s basically no point in someone working on AI risk if they don’t dissolve this specific trauma structure.
Well, or at least make it fully conscious and build their nervous system holding capacity enough that it (a) they can watch it trying to run in real time and (b) they can also reliably stop it from grabbing their inner steering wheel so to speak.
But frankly, for most people it’d be easier just to fully integrate the pain than it would be to develop that level of general nervous system capacity without integrating said pain.
In fact, you can do good and important work while also gradually coming to terms with your emotions, trying to get more grounded, and noticing when you’re making decisions driven by visceral fear and taking steps to fix that.
I agree.
I think we’re focusing on different spots. I’m not sure if we actually disagree.
The all-or-nothing is with respect to recognizing the illusion. If someone can’t even get far enough to notice that their disregulated nervous system is driving an illusion, then what they do is much more likely to create harm than good.
That part I totally stand by.
There’s something of a strawman here in framing what I’m saying as “you need to do the trauma processing first and only then do useful work”. I don’t think you intended it. Just letting you know, it totally lands for me as a strawman.
I am saying that there is some trauma processing (for a person with a system like I’m describing) that absolutely is essential first. But not all of it. I don’t know if that’s possible, or even a coherent idea.
I don’t understand how specifically you think the process of recognizing the illusion is related to the process of healing traumas. But I also object to ideas like “you need to orient towards your fear as an illusion first and only then do useful work”, for roughly the same reasons (in particular, the way it’s all-or-nothing). So I’ll edit my original comment to clarify that this is a more central/less strawmanny objection.
I don’t understand how you think the process of recognizing the illusion is related to the process of healing traumas.
Okay. I’m not sure what to tell you. This lands for me like “I don’t understand how you think turning on the burner is related to the process of cooking the soup.” Um… it just is? I already described the mechanisms, so I think the communication gap is somewhere I don’t see.
I also object to ideas like “you need to orient towards your fear as an illusion first and only then do useful work”…
I never meant to say or even imply that the fear is an illusion.
I was saying that the fear fuels an illusion. And anyone living in such an illusion needs to see through it before they can participate in non-illusion.
You can view that as all-or-nothing and therefore objectionable if you like. That’s note quite what I mean, but it’s not totally wrong. And in this spot I do think there’s an “all-or-nothing” truth: If you don’t see through an illusion you’re in, you can’t consciously participate in reality. That lands as almost tautological to me.
I’ll edit my original comment to clarify that this is a more central/less strawmanny objection.
Okay. I’m not sure what to tell you. This lands for me like “I don’t understand how you think turning on the burner is related to the process of cooking the soup.” Um… it just is? I already described the mechanisms, so I think the communication gap is somewhere I don’t see.
I think you interpreted this as incredulity, whereas I meant it as “I don’t understand the specific links” (e.g. is recognizing the illusion most of the work, or only a small part? What stops you from healing traumas without recognizing the illusion? etc). I’ve edited to clarify.
Oh, no, I didn’t take it as incredulity at all. I’m just honestly not sure why what I’d already said didn’t already explain the relationship between trauma healing and seeing through the illusion.
I guess I can just say it again in shortened form?
For the person design I’m talking about…
There’s a pain inside.
There’s also a kind of mental/emotional program built around the instruction “Distract from the pain.”
Because they can’t actually escape the pain, they project it outward through the mind. Which is to say, they create an illusion powered by the pain.
This causes them to think every glimmer of the pain they do notice is about the external thing.
The antidote is to look directly at the inner pain & dismantle the “Distract from the pain” program.
In practice this requires integrating the pain into consciousness. This is one way of talking about “healing trauma”.
Once that happens, the program doesn’t have a power source anymore.
If that doesn’t happen and the person insists on focusing on doing things in the world, everything they do will be at least partly in service to distraction rather than solving any real problem.
And on the inside they cannot tell the difference between those two without facing the inner pain.
So seeing through the illusion isn’t cognitive basically at all. To me it’s the same thing as trauma processing, for all practical purposes.
Does that clarify anything for you?
What stops you from healing traumas without recognizing the illusion?
Oh, nothing. If you just go around healing traumas willy-nilly, then you might not ever see through any particular illusion like this one if it’s running in you.
Kind of like, generically working on trauma processing in general might or might not help an alcoholic quit drinking. There’s some reason for hope, but it’s possible to get lost in loops of navel-gazing, especially if they never ever even admit to themselves that they have a problem.
But if it’s targeted, the addiction basically doesn’t stand a chance.
I’m not trying to say “Just work on traumas and be Fully Healed™ before working on AI risk.”
I think the word game is in an odd situation here. In game theory parlance, physics is just a game. it’s not a question of whether you’re in a game; it’s what game you interpret yourself to be playing. there are positive-sum games you can interpret yourself to have been playing.
Well, I meant to point at something intuitive, and kind of as a nod to Existential Kink.
I honestly forgot when writing this piece that “game” has special meaning here. Like with game theory.
I just meant to hint at kind of a VR game, or like if you can imagine that the Matrix started out as a game people plugged into but part of the game involves forgetting that you plugged in.
I think there’s a bunch of useful stuff here. In particular, I think that decisions driven by deep-rooted fear are often very counterproductive, and that many rationalists often have “emergency mobilization systems” running in ways which aren’t conducive to good long-term decision-making. I also think that paying attention to bodily responses is a great tool for helping fix this (and in fact was helpful for me in defusing annoyance when reading this post). But I want to push back on the way in which it’s framed in various places as all-or-nothing: exit the game, or keep playing. Get sober, or stay drunk. Hallucination, not real fear.
In fact, you can do good and important work while also gradually coming to terms with your emotions, trying to get more grounded, and noticing when you’re making decisions driven by visceral fear and taking steps to fix that. Indeed, I expect that almost all good and important work throughout history has been done by people who are at various stages throughout that process, rather than people who first dealt with their traumas and only then turned to the work. (EDIT: in a later comment, Valentine says he doesn’t endorse the claim that people should deal with traumas before doing the work, but does endorse the claim that people should recognize the illusion before doing the work. So better to focus on the latter (I disagree with both).)
(This seems more true for concrete research, and somewhat (but less) true for thinking about high-level strategy. In general it seems that rationalists spend way too much of their time thinking about high-level strategic considerations, and I agree with some of Valentine’s reasoning about why this happens. Instead I’d endorse people trying be much more focused on making progress in a few concrete areas, rather than trying to track everything which they think might be relevant to AI risk. E.g. acceleration is probably bad, but it’s fundamentally a second-order effect, and the energy focused on all but the biggest individual instances of acceleration would probably be better used to focus on first-order effects.)
In other words, I want to offer people the affordance to take on board the (many) useful parts of Valentine’s post without needing to buy into the overall frame in which your current concerns are just a game, and your fear is just a manifestation of trauma.
(Relatedly, from my vantage point it seems that “you need to do the trauma processing first and only then do useful work” is a harmful self-propagating meme in a very similar way as “you need to track and control every variable in order for AI to go well”. Both identify a single dominant consideration which requires your full focus and takes precedence over all others. However, I still think that the former is directionally correct for most rationalists, just as the latter is directionally correct for most non-rationalists.)
This. Trauma processing is just as prone to ouroboros-ing as x-risk work, if not more so.
Agreed.
And it’s also not actually relevant to my point.
(Though I understand why it looks relevant.)
Wouldn’t it be relevant in that someone could recognize unproductive, toxic dynamics in their concerns about AI risk as per your point (if I understand you correctly), decide to process trauma first and then get stuck in the same sorts of traps? While “I’m traumatized and need to fix it before I can do anything” may not sound as flashy as “My light cone is in danger from unaligned, high-powered AI and I need to fix that before I can do anything”, it’s just as capable of paralyzing a person, and I speak both from my own past mistakes and from those of multiple friends.
Of course that’s possible. I didn’t mean to dismiss that part.
But… well, as I just wrote to Richard_Ngo:
I do in fact think there’s basically no point in someone working on AI risk if they don’t dissolve this specific trauma structure.
Well, or at least make it fully conscious and build their nervous system holding capacity enough that it (a) they can watch it trying to run in real time and (b) they can also reliably stop it from grabbing their inner steering wheel so to speak.
But frankly, for most people it’d be easier just to fully integrate the pain than it would be to develop that level of general nervous system capacity without integrating said pain.
I agree.
I think we’re focusing on different spots. I’m not sure if we actually disagree.
The all-or-nothing is with respect to recognizing the illusion. If someone can’t even get far enough to notice that their disregulated nervous system is driving an illusion, then what they do is much more likely to create harm than good.
That part I totally stand by.
There’s something of a strawman here in framing what I’m saying as “you need to do the trauma processing first and only then do useful work”. I don’t think you intended it. Just letting you know, it totally lands for me as a strawman.
I am saying that there is some trauma processing (for a person with a system like I’m describing) that absolutely is essential first. But not all of it. I don’t know if that’s possible, or even a coherent idea.
I don’t understand how specifically you think the process of recognizing the illusion is related to the process of healing traumas. But I also object to ideas like “you need to orient towards your fear as an illusion first and only then do useful work”, for roughly the same reasons (in particular, the way it’s all-or-nothing). So I’ll edit my original comment to clarify that this is a more central/less strawmanny objection.
Okay. I’m not sure what to tell you. This lands for me like “I don’t understand how you think turning on the burner is related to the process of cooking the soup.” Um… it just is? I already described the mechanisms, so I think the communication gap is somewhere I don’t see.
I never meant to say or even imply that the fear is an illusion.
I was saying that the fear fuels an illusion. And anyone living in such an illusion needs to see through it before they can participate in non-illusion.
You can view that as all-or-nothing and therefore objectionable if you like. That’s note quite what I mean, but it’s not totally wrong. And in this spot I do think there’s an “all-or-nothing” truth: If you don’t see through an illusion you’re in, you can’t consciously participate in reality. That lands as almost tautological to me.
I didn’t need you to do that. But thanks.
I think you interpreted this as incredulity, whereas I meant it as “I don’t understand the specific links” (e.g. is recognizing the illusion most of the work, or only a small part? What stops you from healing traumas without recognizing the illusion? etc). I’ve edited to clarify.
Oh, no, I didn’t take it as incredulity at all. I’m just honestly not sure why what I’d already said didn’t already explain the relationship between trauma healing and seeing through the illusion.
I guess I can just say it again in shortened form?
For the person design I’m talking about…
There’s a pain inside.
There’s also a kind of mental/emotional program built around the instruction “Distract from the pain.”
Because they can’t actually escape the pain, they project it outward through the mind. Which is to say, they create an illusion powered by the pain.
This causes them to think every glimmer of the pain they do notice is about the external thing.
The antidote is to look directly at the inner pain & dismantle the “Distract from the pain” program.
In practice this requires integrating the pain into consciousness. This is one way of talking about “healing trauma”.
Once that happens, the program doesn’t have a power source anymore.
If that doesn’t happen and the person insists on focusing on doing things in the world, everything they do will be at least partly in service to distraction rather than solving any real problem.
And on the inside they cannot tell the difference between those two without facing the inner pain.
So seeing through the illusion isn’t cognitive basically at all. To me it’s the same thing as trauma processing, for all practical purposes.
Does that clarify anything for you?
Oh, nothing. If you just go around healing traumas willy-nilly, then you might not ever see through any particular illusion like this one if it’s running in you.
Kind of like, generically working on trauma processing in general might or might not help an alcoholic quit drinking. There’s some reason for hope, but it’s possible to get lost in loops of navel-gazing, especially if they never ever even admit to themselves that they have a problem.
But if it’s targeted, the addiction basically doesn’t stand a chance.
I’m not trying to say “Just work on traumas and be Fully Healed™ before working on AI risk.”
I’m saying something much, much more precise.
I think the word game is in an odd situation here. In game theory parlance, physics is just a game. it’s not a question of whether you’re in a game; it’s what game you interpret yourself to be playing. there are positive-sum games you can interpret yourself to have been playing.
Well, I meant to point at something intuitive, and kind of as a nod to Existential Kink.
I honestly forgot when writing this piece that “game” has special meaning here. Like with game theory.
I just meant to hint at kind of a VR game, or like if you can imagine that the Matrix started out as a game people plugged into but part of the game involves forgetting that you plugged in.