When talking to US policymakers, I don’t think there’s a big difference between “causes a national security crisis” and “kills literally everyone.” Worth noting that even though many in the AIS community see a big difference between “99% of people die but civilization restarts” vs. “100% of people die”, IMO this distinction does not matter to most policymakers (or at least matters way less to them).
Of course, in addition to conveying “this is a big deal” you need to convey the underlying threat model. There are lots of ways to interpret “AI causes a national security emergency” (e.g., China, military conflict). “Kills literally everyone” probably leads people to envision a narrower set of worlds.
But IMO even “kills literally everybody” doesn’t really convey the underlying misalignment/AI takeover threat model.
So my current recommendation (weakly held) is probably to go with “causes a national security emergency” or “overthrows the US government” and then accept that you have to do some extra work to actually get them to understand the “AGI--> AI takeover--> Lots of people die and we lose control” model.
My two cents RE particular phrasing:
When talking to US policymakers, I don’t think there’s a big difference between “causes a national security crisis” and “kills literally everyone.” Worth noting that even though many in the AIS community see a big difference between “99% of people die but civilization restarts” vs. “100% of people die”, IMO this distinction does not matter to most policymakers (or at least matters way less to them).
Of course, in addition to conveying “this is a big deal” you need to convey the underlying threat model. There are lots of ways to interpret “AI causes a national security emergency” (e.g., China, military conflict). “Kills literally everyone” probably leads people to envision a narrower set of worlds.
But IMO even “kills literally everybody” doesn’t really convey the underlying misalignment/AI takeover threat model.
So my current recommendation (weakly held) is probably to go with “causes a national security emergency” or “overthrows the US government” and then accept that you have to do some extra work to actually get them to understand the “AGI--> AI takeover--> Lots of people die and we lose control” model.