In which worlds would AI Control prevent significant harm?
When I bring up the issue of AI model security to people working in AI safety, I’m often met with something of the form “yes, this is a problem. It’s important that people work hard on securing AI models. But it doesn’t really affect my work”.
Using AI Control (an area which has recently excited many in the field) as an example, I lay out an argument for why it might not be as effective an agenda as one might think after considering the realities of our cyber security situation.
AI Control concerns itself with models that intentionally try to subvert its developers.
These models are likely to be very generally capable and capable of causing significant harm without countermeasures.
Leading cyber-capable institutions would likely expend significant resources and political capital to steal these models or steal enough insights to reproduce such models.
If the weights or insights are stolen, work on AI control will not prevent these models from causing significant harm.
Current AI developers are not on track to be able to defend against high-priority operations from leading cyber-capable institutions in the coming years.
Therefore, AI control will only be useful in the coming years under one (or more) of these conditions:
Models that scheme are unlikely to be generally capable/dangerous enough to be a high-priority target for leading cyber-capable institutions.
Models that scheme are only developed by actors that can thwart high-priority operations from leading cyber-capable institutions (which precludes current AI developers for at least several years).
AI Control won’t be directly useful in the coming years but it will be indirectly useful to progress the field for when models are developed by actors capable of thwarting top cyber operations.
Even if the model was stolen and caused significant harm, there would still be less harm overall than if the model undergoing AI control measures also broke out. Of course, this argument is much weaker for models capable of existential harm.
Actors that steal the model would also implement strong-enough AI control measures and would not misuse the AI to cause significant harm.
There are of course other arguments against working on AI control. E.g. it may encourage the development and use of models that are capable of causing significant harm. This is an issue if the AI control methods fail or if the model is stolen. So one must be willing to eat this cost or argue that it’s not a large cost when advocating for AI Control work.
This isn’t to say that AI Control isn’t a promising agenda, I just think people need to carefully consider the cases in which their agenda falls down for reasons that aren’t technical arguments about the agenda itself.
I’m also interested to hear takes from those excited by AI Control on which conditions listed in #6 above that they expect to hold (or to otherwise poke holes in the argument).
I’d say a combo of 6B and 6C is my story for AI control working, where AI labor is used to make computer security much better, and importantly security becomes good enough such that even nation-states can’t defeat the security, by a combination of both general security practices becoming better, and AIs becoming good enough at mathematics and coding such that they can generate sound formal proofs without humans being the loop, which is used to defend the most critical infrastructure for AI labs.
This post is probably the best story on how we could get to a state such that AI control is useful:
In general, the hacking capabilities of state actors and the likely involvement of national security when we get closer to AGI feel like significant blind spots of Lesswrong discourse.
(The Hacker and The State by Ben Buchanan is a great book to learn about the former)
jbash seems to think us LessWrong posters have no power to affect global-scale outcomes. I disagree. I believe some of us have quite a bit of power, and should use it before we lose it.
I think one of the priorities of safety-conscious AI developers should be to develop some realistic and downright terrifying demos to show privately to NatSec folks in US/UK/EU. Gotta get that military-grade security in place before we get too much further.
This might be a reason to try to design AI’s to fail-safe and break without controlling units. E.g. before fine-tuning language models to be useful, fine-tune them to not generate useful content without approval tokens generated by a supervisory model.
In which worlds would AI Control prevent significant harm?
When I bring up the issue of AI model security to people working in AI safety, I’m often met with something of the form “yes, this is a problem. It’s important that people work hard on securing AI models. But it doesn’t really affect my work”.
Using AI Control (an area which has recently excited many in the field) as an example, I lay out an argument for why it might not be as effective an agenda as one might think after considering the realities of our cyber security situation.
AI Control concerns itself with models that intentionally try to subvert its developers.
These models are likely to be very generally capable and capable of causing significant harm without countermeasures.
Leading cyber-capable institutions would likely expend significant resources and political capital to steal these models or steal enough insights to reproduce such models.
If the weights or insights are stolen, work on AI control will not prevent these models from causing significant harm.
Current AI developers are not on track to be able to defend against high-priority operations from leading cyber-capable institutions in the coming years.
Therefore, AI control will only be useful in the coming years under one (or more) of these conditions:
Models that scheme are unlikely to be generally capable/dangerous enough to be a high-priority target for leading cyber-capable institutions.
Models that scheme are only developed by actors that can thwart high-priority operations from leading cyber-capable institutions (which precludes current AI developers for at least several years).
AI Control won’t be directly useful in the coming years but it will be indirectly useful to progress the field for when models are developed by actors capable of thwarting top cyber operations.
Even if the model was stolen and caused significant harm, there would still be less harm overall than if the model undergoing AI control measures also broke out. Of course, this argument is much weaker for models capable of existential harm.
Actors that steal the model would also implement strong-enough AI control measures and would not misuse the AI to cause significant harm.
There are of course other arguments against working on AI control. E.g. it may encourage the development and use of models that are capable of causing significant harm. This is an issue if the AI control methods fail or if the model is stolen. So one must be willing to eat this cost or argue that it’s not a large cost when advocating for AI Control work.
This isn’t to say that AI Control isn’t a promising agenda, I just think people need to carefully consider the cases in which their agenda falls down for reasons that aren’t technical arguments about the agenda itself.
I’m also interested to hear takes from those excited by AI Control on which conditions listed in #6 above that they expect to hold (or to otherwise poke holes in the argument).
I’d say a combo of 6B and 6C is my story for AI control working, where AI labor is used to make computer security much better, and importantly security becomes good enough such that even nation-states can’t defeat the security, by a combination of both general security practices becoming better, and AIs becoming good enough at mathematics and coding such that they can generate sound formal proofs without humans being the loop, which is used to defend the most critical infrastructure for AI labs.
This post is probably the best story on how we could get to a state such that AI control is useful:
https://www.lesswrong.com/posts/2wxufQWK8rXcDGbyL/access-to-powerful-ai-might-make-computer-security-radically
In general, the hacking capabilities of state actors and the likely involvement of national security when we get closer to AGI feel like significant blind spots of Lesswrong discourse.
(The Hacker and The State by Ben Buchanan is a great book to learn about the former)
But states seem quite likely to fall under 6e, no?
Quite the opposite, it seems to me, but what you consider “misuse” and “harm” depends on what you value, I suppose.
Maybe we should pick the States we want to be in control, before the States pick for us....
Do you have a preference between Switzerland and North Korea? Cause I sure do.
If we passively let whoever steals the most tech come out as the winner, we might end up with some unpleasant selection effects.
https://www.lesswrong.com/posts/uPi2YppTEnzKG3nXD/nathan-helm-burger-s-shortform?commentId=TwDt9HSh4L3NXFFAF
jbash seems to think us LessWrong posters have no power to affect global-scale outcomes. I disagree. I believe some of us have quite a bit of power, and should use it before we lose it.
I think one of the priorities of safety-conscious AI developers should be to develop some realistic and downright terrifying demos to show privately to NatSec folks in US/UK/EU. Gotta get that military-grade security in place before we get too much further.
This might be a reason to try to design AI’s to fail-safe and break without controlling units. E.g. before fine-tuning language models to be useful, fine-tune them to not generate useful content without approval tokens generated by a supervisory model.