I like the section where you list out specific things you think people should do. (One objection I sometimes hear is something like “I know that [evals/RSPs/if-then plans/misc] are not sufficient, but I just don’t really know what else there is to do. It feels like you either have to commit to something tangible that doesn’t solve the whole problem or you just get lost in a depressed doom spiral.”)
I think your section on suggestions could be stronger by presenting more ambitious/impactful stories of comms/advocacy. I think there’s something tricky about a document that has the vibe “this is the most important issue in the world and pretty much everyone else is approaching it the wrong way” and then pivots to “and the right way to approach it is to post on Twitter and talk to your friends.”
My guess is that you prioritized listing things that were relatively low friction and accessible. (And tbc I do think that the world would be in better shape if more people were sharing their views and contributing to the broad discourse.)
But I think when you’re talking to high-context AIS people who are willing to devote their entire career to work on AI Safety, they’ll be interested in more ambitious/sexy/impactful ways of contributing.
Put differently: Should I really quit my job at [fancy company or high-status technical safety group] to Tweet about my takes, talk to my family/friends, and maybe make some website? Or are there other paths I could pursue?
As I wrote here, I think we have some of those ambitious/sexy/high-impact role models that could be used to make this pitch stronger, more ambitious, and more inspiring. EG:
One possible critique is that their suggestions are not particularly ambitious. This is likely because they’re writing for a broader audience (people who haven’t been deeply engaged in AI safety).
For people who have been deeply engaged in AI safety, I think the natural steelman here is “focus on helping the public/government better understand the AI risk situation.”
There are at least some impactful and high-status examples of this (e.g., Hinton, Bengio, Hendrycks). I think in the last few years, for instance, most people would agree that Hinton/Bengio/Hendrycks have had far more impact in their communications/outreach/policy work than their technical research work.
And it’s not just the famous people– I can think of ~10 junior or mid-career people who left technical research in the last year to help policymakers better understand AI progress and AI risk, and I think their work is likely far more impactful than if they had stayed in technical research. (And I think is true even if I exclude technical people who are working on evals/if-then plans in govt. Like, I’m focusing on people who see their primary purpose as helping the public or policymakers develop “situational awareness”, develop stronger models of AI progress and AI risk, understand the conceptual arguments for misalignment risk, etc.)
I’d also be curious to hear what your thoughts are on people joining government organizations (like the US AI Safety Institute, UK AI Safety Institute, Horizon Fellowship, etc.) Most of your suggestions seem to involve contributing from outside government, and I’d be curious to hear more about your suggestions for people who are either working in government or open to working in government.
Last minute we ended up cutting a section at the end of the document called “how does this [referring to civics, communications, coordination] all add up to preventing extinction?” It was an attempt to address the thing you’re pointing at here:
I think there’s something tricky about a document that has the vibe “this is the most important issue in the world and pretty much everyone else is approaching it the wrong way” and then pivots to “and the right way to approach it is to post on Twitter and talk to your friends.”
Sadly we didn’t feel we could get the point across well enough to make our timing cutoff for v1. A quick attempt at the same answer here (where higher context might make it easier to convey the point):
One way someone could be asking “does this all add up” is “are we going to survive?” And to that, our answer is mostly “hmm, that’s not really a question we think about much. Whether we’re going to make it or not is a question of fact, not opinion. We’re just trying our best to work on what we think is optimal.”
The other way someone could be asking “does this all add up” is “is this really a good plan?” That’s a great question—now we’re talking strategy.
There are of course huge things that need to be done. A number of the authors support what’s written in A Narrow Path, which offers very ambitious projects for AI policy. This is one good way to do strategy: start with the “full plan” and then use that as your map. If your plan can’t keep us safe from superintelligence even in the ideal case where everything is implemented, then you need a new plan. (This is one of many of our RSP concerns—what is the full plan? Most everything after “we detect the dangerous thing” still seems to require the level of intervention described in A Narrow Path)
Communication, coordination, and civics straightforwardly don’t add up to A Narrow Path’s suggestions. However, they are bottlenecks. (Why we think this is communicated somewhat in the doc, but there’s a lot more to say on this). This is another good way to do strategy: look at things that are required in any good plan, and optimize for those. We don’t see a winning world where AGI risks are not a global common knowledge, with people aware and concerned and acting at a scale far larger scale than today.
Where A Narrow Path does backprop from what’s needed to actually stop superintelligence from being built, this doc presents more of “what can we do immediately, today.” We try to tie the “What we can do immediately” to those bottlenecks that we think are needed in any plan.
And yes—It was written more with a low-context AIS person in mind, and most of the high-context people we try to redirect towards “hey, reach out to us if you’d like to be more deeply involved.” I think v2 should include more suggestions for bigger projects, that people with more context can pursue. Would love to hear your (or others’) views on good projects.
Also, great comment about the people who have done the most on communication so far. I really commend their efforts and writing something about this is definitely something v2 can include.
On joining government organizations… I’m just speaking for myself on this one as I think my coauthors have different views on governance than myself. Yes—this is good and necessary. Two caveats:
Be willing to go against the grain. It seems the default path right now is for governments to support the same “reactive framework” that AGI companies are pushing. I’m worried about this, and I think we need people in government positions and advising them that are much more frank about the risks and unwilling to go for “convenient” solutions that fit the overton window. If the necessary safety regulations don’t fit the current overton window, then the overton window has to change, not the regulation. Huge props to CAIS for SB1047 and whatever future bill efforts follow from them or others.
Be willing to help. Lots of people in government do care, and simply don’t know what’s going on. Try to be helpful to them before assuming they’re antagonistic to x-risk. I’ve met lots of government people who are very amenable to a “hey, I’d be super happy to talk you through the technical details of AI, and explain why some people are worried about x-risk.” Non-threatening, non-asking-for-something approaches really work.
More to say later—to this and other comments in the thread. For now, taking the weekend to get a bit of rest :)
I like the section where you list out specific things you think people should do. (One objection I sometimes hear is something like “I know that [evals/RSPs/if-then plans/misc] are not sufficient, but I just don’t really know what else there is to do. It feels like you either have to commit to something tangible that doesn’t solve the whole problem or you just get lost in a depressed doom spiral.”)
I think your section on suggestions could be stronger by presenting more ambitious/impactful stories of comms/advocacy. I think there’s something tricky about a document that has the vibe “this is the most important issue in the world and pretty much everyone else is approaching it the wrong way” and then pivots to “and the right way to approach it is to post on Twitter and talk to your friends.”
My guess is that you prioritized listing things that were relatively low friction and accessible. (And tbc I do think that the world would be in better shape if more people were sharing their views and contributing to the broad discourse.)
But I think when you’re talking to high-context AIS people who are willing to devote their entire career to work on AI Safety, they’ll be interested in more ambitious/sexy/impactful ways of contributing.
Put differently: Should I really quit my job at [fancy company or high-status technical safety group] to Tweet about my takes, talk to my family/friends, and maybe make some website? Or are there other paths I could pursue?
As I wrote here, I think we have some of those ambitious/sexy/high-impact role models that could be used to make this pitch stronger, more ambitious, and more inspiring. EG:
I’d also be curious to hear what your thoughts are on people joining government organizations (like the US AI Safety Institute, UK AI Safety Institute, Horizon Fellowship, etc.) Most of your suggestions seem to involve contributing from outside government, and I’d be curious to hear more about your suggestions for people who are either working in government or open to working in government.
Thanks Akash for this substantive reply!
Last minute we ended up cutting a section at the end of the document called “how does this [referring to civics, communications, coordination] all add up to preventing extinction?” It was an attempt to address the thing you’re pointing at here:
Sadly we didn’t feel we could get the point across well enough to make our timing cutoff for v1. A quick attempt at the same answer here (where higher context might make it easier to convey the point):
One way someone could be asking “does this all add up” is “are we going to survive?” And to that, our answer is mostly “hmm, that’s not really a question we think about much. Whether we’re going to make it or not is a question of fact, not opinion. We’re just trying our best to work on what we think is optimal.”
The other way someone could be asking “does this all add up” is “is this really a good plan?” That’s a great question—now we’re talking strategy.
There are of course huge things that need to be done. A number of the authors support what’s written in A Narrow Path, which offers very ambitious projects for AI policy. This is one good way to do strategy: start with the “full plan” and then use that as your map. If your plan can’t keep us safe from superintelligence even in the ideal case where everything is implemented, then you need a new plan. (This is one of many of our RSP concerns—what is the full plan? Most everything after “we detect the dangerous thing” still seems to require the level of intervention described in A Narrow Path)
Communication, coordination, and civics straightforwardly don’t add up to A Narrow Path’s suggestions. However, they are bottlenecks. (Why we think this is communicated somewhat in the doc, but there’s a lot more to say on this). This is another good way to do strategy: look at things that are required in any good plan, and optimize for those. We don’t see a winning world where AGI risks are not a global common knowledge, with people aware and concerned and acting at a scale far larger scale than today.
Where A Narrow Path does backprop from what’s needed to actually stop superintelligence from being built, this doc presents more of “what can we do immediately, today.” We try to tie the “What we can do immediately” to those bottlenecks that we think are needed in any plan.
And yes—It was written more with a low-context AIS person in mind, and most of the high-context people we try to redirect towards “hey, reach out to us if you’d like to be more deeply involved.” I think v2 should include more suggestions for bigger projects, that people with more context can pursue. Would love to hear your (or others’) views on good projects.
Also, great comment about the people who have done the most on communication so far. I really commend their efforts and writing something about this is definitely something v2 can include.
On joining government organizations… I’m just speaking for myself on this one as I think my coauthors have different views on governance than myself. Yes—this is good and necessary. Two caveats:
Be willing to go against the grain. It seems the default path right now is for governments to support the same “reactive framework” that AGI companies are pushing. I’m worried about this, and I think we need people in government positions and advising them that are much more frank about the risks and unwilling to go for “convenient” solutions that fit the overton window. If the necessary safety regulations don’t fit the current overton window, then the overton window has to change, not the regulation. Huge props to CAIS for SB1047 and whatever future bill efforts follow from them or others.
Be willing to help. Lots of people in government do care, and simply don’t know what’s going on. Try to be helpful to them before assuming they’re antagonistic to x-risk. I’ve met lots of government people who are very amenable to a “hey, I’d be super happy to talk you through the technical details of AI, and explain why some people are worried about x-risk.” Non-threatening, non-asking-for-something approaches really work.
More to say later—to this and other comments in the thread. For now, taking the weekend to get a bit of rest :)