I like this idea. At AI Impacts we were discussing something similar: having “fire drills” where we spend a week (or even just a day) pretending that a certain scenario has happened, e.g. “DeepMind just announced they have a turing-test-passing system and will demo it a week from now; we’ve got two journalists asking us for interviews and need to prep for the emergency meeting with the AI safety community tonight at 5.” We never got around to testing out such a drill but I think variants on this idea are worth exploring. Inspired by what you said, perhaps we could have “snap drills” where suddenly we take our goals for the next two months and imagine that they need to be accomplished in a week instead, and see how much we can do. (Additionally, ideas like this seem like they would have bonus effects on morale, teamwork, etc.)
I don’t know what is entailed in cultivating that virtue. Perhaps meditation? Maybe testing one’s self at literal risk to one’s life?
This virtue is extremely important to militaries. Does any military use meditation as part of its training? I would guess that the training given to medics and officers (soldiers for whom clear thinking is especially important) might have some relevant lessons. Then again, maybe the military deals with this primarily by selecting the right sort of people rather than taking arbitrary people and training them. If so, perhaps we should look into applying similar selection methods in our own organizations to identify people to put in charge when the time comes.
Perhaps it would be good to have an Official List of all the AI safety strategies, so that whatever rationale people give for why this AI is safe can be compared to the list. (See this prototype list.)
Perhaps it would be good to have an Official List of all the AI safety problems, so that whatever rationale people give for why this AI is safe can be compared to the list, e.g. “OK, so how does it solve outer alignment? What about mesa-optimizers? What about the malignity of the universal prior? I see here that your design involves X; according to the Official List, that puts it at risk of developing problems Y and Z...” (See this prototype list.)
Perhaps it would be good to have various important concepts and arguments re-written with an audience of skeptical and impatient AI researchers in mind, rather than the current audience of friends and LessWrong readers.
Thinking afresh, here’s another idea: I have a sketch of a blog post titled “What Failure Feels Like.” The idea is to portray a scenario of doom in general, abstract terms (like Paul’s post does, as opposed to writing a specific, detailed story) but with a focus on how it feels to us AI-risk-reducers, rather than focusing on what the world looks like in general or what’s going on inside the AIs. I decided it would be depressing and not valuable to write. However, maybe it would be valuable as a thing people could read to help emotionally prepare/steel themselves for the time when they “are confronted with the stark reality of how doomed we are.” IDK.
I guess overall my favorite idea is to just periodically spend time thinking about what you’d do if you found out that takeoff was happening soon. E.g. “Deepmind announces turing-test system” or “We learn of convincing roadmap to AGI involving only 3 OOMs more compute” or “China unveils project to spend +7 OOMs on a single training run by 2030, with lesser training runs along the way” I think that the exercise of thinking about near-term scenarios and then imagining what we’d do in response will be beneficial even on long timelines, but certainly super beneficial on short timelines (even if, as is likely, none of the scenarios we imagine come to pass).
I was coming to say something similar [edited to add: about communication skills.] I don’t know much about this field, but one comparison that comes to mind is Ignaz Semmelweis who discovered that hand-cleaning prevented hospital deaths, but let his students write it up instead of trying to convince his colleagues more directly. The message got garbled, his colleagues thought he was a crank, and continental Europe’s understanding of germ theory was delayed by 60 years as a result.
Regarding “Staying grounded and stable in spite of the stakes”: I think it might be helpful to unpack the vritue/skill(s) involved according to the different timescales at which emergencies unfold.
For example:
1. At the time scale of minutes or hours, there is a virtue/skill of “staying level headed in a situation of accute crisis”. This is the sort of skill you want your emergency doctor or firefighter to have. (When you pointed to the military, I think you in part pointed to this scale but I assume not only.)
From talking to people who do or did jobs like this, a typical pattern seems to be that some types of people when in siutations like this basically “freeze” and others basically move into a mode of “just functioning”. There might be some margin for practice here (maybe you freeze the first time around and are able to snap out of the freeze the second time around, and after that, you can “remember” what it feels like to shift into funcitoning mode ever after) but, according to the “common wisdom” in these prfoessions (as I undestand it), mostly people seem to fall in one or the other category.
The sort of practice that I see being helpful here is a) overtraining on whatever skill you will need in the moment (e.g. imagine the emergency doctor) such that you can hand over most cognitive work to your autopilot once the emergency occurs; and b) train the skill of switching from freeze into high-functioning mode. I would expect “drill-type practices” are the most abt to get at that, but as noted above I don’t know how large the margin for improvement is. (A subtlety here: there seems to be a massive difference between “being the first person to switch in to funcitoning mode”, vs “switching into functioning mode after (literally or metaphorically speaking) someone screamed at your face to get moving”. (Thinking of the military here.))
All that said, I don’t feel particularly excited for people to start doing a bunch of drill practice or the like. I think there are possible extreme scenarios of “narrow hingy moments” that will involve this skill but overall this doesn’t seem to me not to be the thing that is most needed/with highest EV.
(Probably also worth putting some sort of warning flag here: genuinly high-intensity situations can be harmful to people’s psychy so one should be very cautious about experimenting with things in this space.)
2. Next, there might be a related virtue/skill at the timescale of weeks and months. I think the pandemic, especially from ~March to May/June is an excellent example of this, and was also an excellent learning opportunities for people involved in some time-sensitive covid-19 problem. I definitely think I’ve gained some gears on what a genuin (i.e. highly stakey) 1-3 month sprint involves, and what challenges and risks are invovled for you as an “agent” who is trying to also protect their agency/ability to think and act (though I think others have learnt and been stress-tested much more than I have).
Personally, my sense is that this is “harder” than the thing in 1., because you can’t rely on your autopilot much, and this makes things feel more like an adaptive rather than technical problem (where the latter is aproblem where the solution is basically clear, you just have to do it; and the latter is a problem most of the work needed is in figuring out the solution, not so much (necessarily) in executing it.)
One difficulty is that this skill/virtue involves managing your energy not only spending it well. Knowing yourself and hoy your energy and motivation structures work—and in particular how they work in extreme scenarios—seems very important. I can see how people who have meditated a lot have gained valuable skills here. I don’t think it’s th eonly way to get these skills, and I expect the thing that is paying off here is more “being able to look back on years of meditaton practice and the ways this has rewired one’s brain in some deep sense” rather than “benefits from having a routine to meditate” or something like this.
During the first couple of COVID-19 months, I was also surprised how “doing well at this” was more a question of collective rationality than I would have thought (by collective rationality I mean things like: ability to communciate effectively, ability to mobilise people/people with the right skills, abilty to delegate work effectively). There is still a large individual component of “staying on top of it all/keeping the horizon in sight” such that you are able to make hard decisoins (which you will be faced with en masse).
I think it could be really good to collect lessons learnt from the folks invovled in some EA/rationlaist-adjacent COVID-19 projects.
3. The scale of ~(a few) years seems quite similar in type to 2. The main thing that I’d want to add here is that the challenge of dealing with stronguncertainty while the stakes are massive can be very psychologically challenge. I do think meditation and related practices can be helpful in dealing with that in a way that is both grounded and not flinching from the truth.
I find myself wondering whether the miliatry does anything to help soldiers prepare for the act of “going to war” where the posisbility of death is extremely real. I imaigne they must do things to support people in this process. It’s not exactly the same but there certainly are parallels with what we want.
This [2019] winter, Army infantry soldiers at Schofield Barracks in Hawaii began using mindfulness to improve shooting skills — for instance, focusing on when to pull the trigger amid chaos to avoid unnecessary civilian harm.
The British Royal Navy has given mindfulness training to officers, and military leaders are rolling it out in the Army and Royal Air Force for some officers and enlisted soldiers. The New Zealand Defence Force recently adopted the technique, and military forces of the Netherlands are considering the idea, too.
This week, NATO plans to hold a two-day symposium in Berlin to discuss the evidence behind the use of mindfulness in the military.
A small but growing group of military officials support the techniques to heal trauma-stressed veterans, make command decisions and help soldiers in chaotic battles.
“I was asked recently if my soldiers call me General Moonbeam,” said Maj. Gen. Piatt, who was director of operations for the Army and now commands its 10th Mountain Division. “There’s a stereotype this makes you soft. No, it brings you on point.”
The approach, he said, is based on the work of Amishi Jha, an associate professor of psychology at the University of Miami. She is the senior author of a paper published in December about the training’s effectiveness among members of a special operations unit.
The findings, which build on previous research showing improvements among soldiers and professional football players trained in mindfulness, are significant in part because members of the special forces are already selected for their ability to focus. The fact that even they saw improvement speaks to the power of the training, Dr. Jha said. [...]
Mr. Boughton has thought about whether mindfulness is anathema to conflict. “The purists would say that mindfulness was never developed for war purpose,” he said.
What he means is that mindfulness is often associated with peacefulness. But, he added, the idea is to be as faithful to compassionate and humane ideals as possible given the realities of the job.
Maj. Gen. Piatt underscored that point, describing one delicate diplomatic mission in Iraq that involved meeting with a local tribal leader. Before the session, he said, he meditated in front of a palm tree, and found himself extremely focused when the delicate conversation took place shortly thereafter.
“I was not taking notes. I remember every word she was saying. I wasn’t forming a response, just listening,” he said. When the tribal leader finished, he said, “I talked back to her about every single point, had to concede on some. I remember the expression on her face: This is someone we can work with.”
Hmmm, if this is the most it’s been done, then that counts as a No in my book. I was thinking something like “Ah yes, the Viet Cong did this for most of the war, and it’s now standard in both the Vietnamese and Chinese armies.” Or at least “Some military somewhere has officially decided that this is a good idea and they’ve rolled it out across a large portion of their force.”
Thanks, this is a great thing to be thinking about and a good list of ideas!
Public speaking skills, persuasion skills, debate skills, etc.
I like this idea. At AI Impacts we were discussing something similar: having “fire drills” where we spend a week (or even just a day) pretending that a certain scenario has happened, e.g. “DeepMind just announced they have a turing-test-passing system and will demo it a week from now; we’ve got two journalists asking us for interviews and need to prep for the emergency meeting with the AI safety community tonight at 5.” We never got around to testing out such a drill but I think variants on this idea are worth exploring. Inspired by what you said, perhaps we could have “snap drills” where suddenly we take our goals for the next two months and imagine that they need to be accomplished in a week instead, and see how much we can do. (Additionally, ideas like this seem like they would have bonus effects on morale, teamwork, etc.)
This virtue is extremely important to militaries. Does any military use meditation as part of its training? I would guess that the training given to medics and officers (soldiers for whom clear thinking is especially important) might have some relevant lessons. Then again, maybe the military deals with this primarily by selecting the right sort of people rather than taking arbitrary people and training them. If so, perhaps we should look into applying similar selection methods in our own organizations to identify people to put in charge when the time comes.
In this post I discuss some:
Thinking afresh, here’s another idea: I have a sketch of a blog post titled “What Failure Feels Like.” The idea is to portray a scenario of doom in general, abstract terms (like Paul’s post does, as opposed to writing a specific, detailed story) but with a focus on how it feels to us AI-risk-reducers, rather than focusing on what the world looks like in general or what’s going on inside the AIs. I decided it would be depressing and not valuable to write. However, maybe it would be valuable as a thing people could read to help emotionally prepare/steel themselves for the time when they “are confronted with the stark reality of how doomed we are.” IDK.
I guess overall my favorite idea is to just periodically spend time thinking about what you’d do if you found out that takeoff was happening soon. E.g. “Deepmind announces turing-test system” or “We learn of convincing roadmap to AGI involving only 3 OOMs more compute” or “China unveils project to spend +7 OOMs on a single training run by 2030, with lesser training runs along the way” I think that the exercise of thinking about near-term scenarios and then imagining what we’d do in response will be beneficial even on long timelines, but certainly super beneficial on short timelines (even if, as is likely, none of the scenarios we imagine come to pass).
I was coming to say something similar [edited to add: about communication skills.]
I don’t know much about this field, but one comparison that comes to mind is Ignaz Semmelweis who discovered that hand-cleaning prevented hospital deaths, but let his students write it up instead of trying to convince his colleagues more directly. The message got garbled, his colleagues thought he was a crank, and continental Europe’s understanding of germ theory was delayed by 60 years as a result.
Which bit was this replying to?
oh right, about the public speaking / communication type skills.
Regarding “Staying grounded and stable in spite of the stakes”:
I think it might be helpful to unpack the vritue/skill(s) involved according to the different timescales at which emergencies unfold.
For example:
1. At the time scale of minutes or hours, there is a virtue/skill of “staying level headed in a situation of accute crisis”. This is the sort of skill you want your emergency doctor or firefighter to have. (When you pointed to the military, I think you in part pointed to this scale but I assume not only.)
From talking to people who do or did jobs like this, a typical pattern seems to be that some types of people when in siutations like this basically “freeze” and others basically move into a mode of “just functioning”. There might be some margin for practice here (maybe you freeze the first time around and are able to snap out of the freeze the second time around, and after that, you can “remember” what it feels like to shift into funcitoning mode ever after) but, according to the “common wisdom” in these prfoessions (as I undestand it), mostly people seem to fall in one or the other category.
The sort of practice that I see being helpful here is a) overtraining on whatever skill you will need in the moment (e.g. imagine the emergency doctor) such that you can hand over most cognitive work to your autopilot once the emergency occurs; and b) train the skill of switching from freeze into high-functioning mode. I would expect “drill-type practices” are the most abt to get at that, but as noted above I don’t know how large the margin for improvement is. (A subtlety here: there seems to be a massive difference between “being the first person to switch in to funcitoning mode”, vs “switching into functioning mode after (literally or metaphorically speaking) someone screamed at your face to get moving”. (Thinking of the military here.))
All that said, I don’t feel particularly excited for people to start doing a bunch of drill practice or the like. I think there are possible extreme scenarios of “narrow hingy moments” that will involve this skill but overall this doesn’t seem to me not to be the thing that is most needed/with highest EV.
(Probably also worth putting some sort of warning flag here: genuinly high-intensity situations can be harmful to people’s psychy so one should be very cautious about experimenting with things in this space.)
2. Next, there might be a related virtue/skill at the timescale of weeks and months. I think the pandemic, especially from ~March to May/June is an excellent example of this, and was also an excellent learning opportunities for people involved in some time-sensitive covid-19 problem. I definitely think I’ve gained some gears on what a genuin (i.e. highly stakey) 1-3 month sprint involves, and what challenges and risks are invovled for you as an “agent” who is trying to also protect their agency/ability to think and act (though I think others have learnt and been stress-tested much more than I have).
Personally, my sense is that this is “harder” than the thing in 1., because you can’t rely on your autopilot much, and this makes things feel more like an adaptive rather than technical problem (where the latter is aproblem where the solution is basically clear, you just have to do it; and the latter is a problem most of the work needed is in figuring out the solution, not so much (necessarily) in executing it.)
One difficulty is that this skill/virtue involves managing your energy not only spending it well. Knowing yourself and hoy your energy and motivation structures work—and in particular how they work in extreme scenarios—seems very important. I can see how people who have meditated a lot have gained valuable skills here. I don’t think it’s th eonly way to get these skills, and I expect the thing that is paying off here is more “being able to look back on years of meditaton practice and the ways this has rewired one’s brain in some deep sense” rather than “benefits from having a routine to meditate” or something like this.
During the first couple of COVID-19 months, I was also surprised how “doing well at this” was more a question of collective rationality than I would have thought (by collective rationality I mean things like: ability to communciate effectively, ability to mobilise people/people with the right skills, abilty to delegate work effectively). There is still a large individual component of “staying on top of it all/keeping the horizon in sight” such that you are able to make hard decisoins (which you will be faced with en masse).
I think it could be really good to collect lessons learnt from the folks invovled in some EA/rationlaist-adjacent COVID-19 projects.
3. The scale of ~(a few) years seems quite similar in type to 2. The main thing that I’d want to add here is that the challenge of dealing with strong uncertainty while the stakes are massive can be very psychologically challenge. I do think meditation and related practices can be helpful in dealing with that in a way that is both grounded and not flinching from the truth.
I find myself wondering whether the miliatry does anything to help soldiers prepare for the act of “going to war” where the posisbility of death is extremely real. I imaigne they must do things to support people in this process. It’s not exactly the same but there certainly are parallels with what we want.
. Yes, e.g.
Hmmm, if this is the most it’s been done, then that counts as a No in my book. I was thinking something like “Ah yes, the Viet Cong did this for most of the war, and it’s now standard in both the Vietnamese and Chinese armies.” Or at least “Some military somewhere has officially decided that this is a good idea and they’ve rolled it out across a large portion of their force.”