I think many members of the policy community feel like ARA is “weird” and therefore don’t want to bring it up. It’s much tamer to talk about CBRN threats and bioweapons. It also requires less knowledge and general competence– explaining ARA and autonomous systems risks is difficult, you get more questions, you’re more likely to explain something poorly, etc.
Historically, there was also a fair amount of gatekeeping, where some of the experienced policy people were explicitly discouraging people from being explicit about AGI threat models (this still happens to some degree, but I think the effect is much weaker than it was a year ago.)
With all this in mind, I currently think raising awareness about ARA threat models and AI R&D threat models is one of the most important things for AI comms/policy efforts to get right.
In the status quo, even if the evals go off, I don’t think we have laid the intellectual foundation required for policymakers to understand why the evals are dangerous. “Oh interesting– an AI can make copies of itself? A little weird but I guess we make copies of files all the time, shrug.” or “Oh wow– AI can help with R&D? That’s awesome– seems very exciting for innovation.”
I do think there’s a potential to lay the intellectual foundation before it’s too late, and I think many groups are starting to be more direct/explicit about the “weirder” threat models. Also, I think national security folks have more of a “take things seriously and worry about things even if there isn’t clear empirical evidence yet” mentality than ML people. And I think typical policymakers fall somewhere in between.
We need national security people across the world to be speaking publicly about this, otherwise the general discussion on threaats and risks of AI remains gravely incomplete and bias
The problem is that as usual people will worry that the NatSec guys are using the threat to try to slip us the pill of additional surveillance and censorship for political purposes—and they probably won’t be entirely wrong. We keep undermining our civilizational toolset by using extreme measures for trivial partisan stuff and that reduces trust.
I think many members of the policy community feel like ARA is “weird” and therefore don’t want to bring it up. It’s much tamer to talk about CBRN threats and bioweapons. It also requires less knowledge and general competence– explaining ARA and autonomous systems risks is difficult, you get more questions, you’re more likely to explain something poorly, etc.
Historically, there was also a fair amount of gatekeeping, where some of the experienced policy people were explicitly discouraging people from being explicit about AGI threat models (this still happens to some degree, but I think the effect is much weaker than it was a year ago.)
With all this in mind, I currently think raising awareness about ARA threat models and AI R&D threat models is one of the most important things for AI comms/policy efforts to get right.
In the status quo, even if the evals go off, I don’t think we have laid the intellectual foundation required for policymakers to understand why the evals are dangerous. “Oh interesting– an AI can make copies of itself? A little weird but I guess we make copies of files all the time, shrug.” or “Oh wow– AI can help with R&D? That’s awesome– seems very exciting for innovation.”
I do think there’s a potential to lay the intellectual foundation before it’s too late, and I think many groups are starting to be more direct/explicit about the “weirder” threat models. Also, I think national security folks have more of a “take things seriously and worry about things even if there isn’t clear empirical evidence yet” mentality than ML people. And I think typical policymakers fall somewhere in between.
We need national security people across the world to be speaking publicly about this, otherwise the general discussion on threaats and risks of AI remains gravely incomplete and bias
The problem is that as usual people will worry that the NatSec guys are using the threat to try to slip us the pill of additional surveillance and censorship for political purposes—and they probably won’t be entirely wrong. We keep undermining our civilizational toolset by using extreme measures for trivial partisan stuff and that reduces trust.