OpenAI already have this in their charter:
We are concerned about late-stage AGI development becoming a competitive race without time for adequate safety precautions. Therefore, if a value-aligned, safety-conscious project comes close to building AGI before we do, we commit to stop competing with and start assisting this project. We will work out specifics in case-by-case agreements, but a typical triggering condition might be “a better-than-even chance of success in the next two years.”
Similar opinion here, also noting they didn’t run red-teaming and persuasion evals on the actually-final-version:
https://x.com/teortaxesTex/status/1885401111659413590
I think Control has similar problems to RLHF, where both might fail to generalize out of distribution in similar ways.
This seems important because Control has a central goal of being a fallback for that kind of failure mode.
I want to share my thoughts, including something nice Control does about this problem that I think RLHF could easily incorporate (investigate failures instead of always training against them).
What do I mean by Control generalizing out of distribution:
Our Control experiment might involve giving a model 100 leetcode problems, and we hope to show that (even with help) the model can’t successfully insert a back door without being caught by our Control setup[1].
Will this generalize to other leetcode problems? Probably.
How about more realistic coding tasks that involve confusing legacy code? Who knows.
How about more open-ended research tasks, like “here’s Ryan’s pitch for a faking-alignment paper, fill in the details and run the experiment”? I, eh, doubt it? Remember—we’re dealing with an AI that might be trying to scheme and mess up our research on purpose.
If we tested whether the AI can leak sensitive data without being caught, will it generalize to other bad things it might try to do?
And so on
Control’s soft guarantees could be somewhat achieved using RLHF
I’m writing this both to emphasise the similarity, but also to suggest nice things we could do with RLHF:
We generate scenarios where an AI might scheme or not scheme.
We might use a similar Control setup (“adding back doors to code”?), but the monitoring (“control protocol”) doesn’t have to be cheap+fast+scalable enough to run in production.
Given e.g 100 scenarios, use RLHF to train on a fraction of them (e.g 20), and check if the RLHF generalized to the rest.
If it didn’t generalize, pretend you caught the AI red handed (investigate what happened, publish it publicly, don’t just train against these scenarios)
In general, I’d be excited about labs publishing a naive metric like “what fraction of our RLHF results were needed in order to generalize to the rest”, as well as examples for scenarios that the AI doesn’t seem to generalize to unless you train against them specifically.
For similar reasons, I’d be excited about “we got our Control protocol working on 20% of the scenarios and it successfully generalized to the other 80%” over “we fixed our Control protocol until it worked[2] on 100% of the scenarios we had”.
Oh, and for both RLHF and Control, I think we should be very interested in “what is the distribution they used”.
Replies I got to this:
“We will only use the Controlled AI to write pretty normal code, so the deployment setup is in-distribution”
Me: “So how would Control help Safety research more than Capabilities? Seems like labs will try to extract AI R&D work regardless.”
“Control will let us run an AI that is smart[3] enough to cause significant damage to humanity, and extracting coding-work from such an AI is valuable”
Me: “That scenario implies an AI that can scheme in many creative ways. The Control scenario above might have (metaphorically) found the “backdoored code” neuron. That doesn’t seem enough.”
My steelman of Redwood:
“Yes, the distribution of scheming examples is super important, here are 2 posts we wrote about it and you missed, and we’ll publish 5 more next week with 14 more distributions of tasks relevant for the kind of alignment research we plan to do with a Controlled AI” (seems like Redwood always have a post directly addressing my concerns..)
“We need to start somewhere, and it’s easy to start with backdoored code or with a simple scheming objective like making an HTTP request”
“It’s useful to focus on scheming (whether using Control or RLHF) because that leads to other great assumptions”
“People are already implementing those RLHF suggestions, here are 20 arxiv papers about it that you missed”
“Control gives us less-capabilities with more-failing-loudly, which is a tradeoff we like”
“We are trying to have Control give (partial) guarantees that are pretty uncorrelated with RLHF, just as another layer of safety, because it’s the best idea we have so far, not because we actually think this will prevent the AI from scheming in a somewhat general way”
An opinion from a former lawyer
[disclaimers: they’re not an anti trust lawyer and definitely don’t take responsibility for this opinion, nor do I. This all might maybe be wrong and we need to speak to an actual anti-trust lawyer to get certainty. I’m not going to put any more disclaimers here, I hope I’m not also misremembering something]
So,
Having someone from the U.S government sign that they won’t enforce anti trust laws isn’t enough (even if the president signs), because the (e.g) president might change their mind, or the next president might enforce it retroactively. This is similar to the current situation with Tiktok where Trump said he wouldn’t enforce the law that prevents Google from having Tiktok on their app store, but Google still didn’t put Tiktok back, probably because they’re afraid that someone will change their mind and retroactively enforce the law
I asked if the government (e.g president) could sign “we won’t enforce this, and if we change our mind we’ll give a 3 month notice”.
The former-lawyer’s response was to consider whether, in a case the president would change their mind immediately, this signature would hold up in court. He thinks that not, but couldn’t remember an example of something similar happening (which seems relevant)
If the law changes (for example, to exclude this letter), that works
(but it’s hard to pass such changes through congress)
If the letter is conditional on the law changing, that seems ok
My interpretation of this:
It seems probably possible to find a solution where signing this letter is legal, but we’d have to consult with an anti-trust lawyer.
[reminder that this isn’t legal advice, isn’t confident, is maybe misremembered, and so on]
To the extent you think Anthropic is a good actor, you shouldn’t be pressuring good actors like them to take actions that will make them differentially less competitive than worse actors
I think an important part of how one becomes (and stays) a good actor is by being transparent about things like this.
Anthropic could at least avoid making substantially misleading statements
Yes. But also, I’m afraid that Anthropic might solve this problem by just making less statements (which seems bad). Still Yes
Hey,
In the article, you measured the MONA setup against a pure RL agent.
I’m curious about measuring MONA against the overseer-AI picking the next move directly[1]: The overseer-AI probably won’t[2] reward hack more than the MONA setup, but it’s unclear to me if it would also have worse performance.
I could imagine discovering the myopic MONA agent converging on
Picking whatever the overseer-AI would pick (since those actions would seem “obviously correct” to the overseer AI and result in the most reward)
Picking actions that seem impressive but are useless (bad! we could get better performance, still with ~no reward hacking, by asking the overseer directly[1])
Picking clever useful actions that the overseer wouldn’t have thought of (win!)
I’d give the overseer-AI scaffolding to make it more comparable to the MONA setup, such as the ability to observe the consequences of its own actions and roll them back if they seem bad.
MONA is trying to use the overseer-AI to put an upper bound on how much reward hacking happens, if I understood correctly. If anyone runs this experiment, I’d still validate that assumption, since it seems easy (the paper already has a way to measure reward hacking).
nit: I wouldn’t use a prediction market as an overseer because markets are often uninterpretable to humans, which would miss some of the point[1].
“we show how to get agents whose long-term plans follow strategies that humans can predict”. But maybe no single human actually understands the strategy. Or maybe the traders are correctly guessing that the model’s steps will somehow lead to whatever is defined as a “good outcome”, even if they don’t understand how, which has similar problems to the RL reward from the future that you’re trying to avoid.
For a simple task like booking a restaurant, we could just ask the (frozen) overseer-AI to pick[1] actions, no?
The interesting application MONA seems to be when the myopic RL agent is able to produce better suggestions than the overseer
Edit: I elaborated
Plus maybe let the overseer observe the result and say “oops” and roll back that action, if we can implement a rollback in this context
US Gov isn’t likely to sign: Seems right.
OpenAI isn’t likely to sign: Seems right.
Still, I think this letter has value, especially if it has “P.S. We’re making this letter because we think if everyone keeps racing then there’s a noticable risk of everyone dying. We think it would be worse if only we stop, but having everyone stop would be the safest, and we think this opinion of ours should be known publicly”
Dario said (Nov 2024):
“I never liked those words [P(DOOM)], I think they’re kinda wired, my view is we should measure risks as they come up, and in the meantime we should get all the economic benefits that we could get. And we should find ways to measure the risks that are effective, but are minimally disruptive to the amazing economic process that we see going on now that the last thing we want to do is slow down” [lightly edited for clarity, bold is mine]
If he doesn’t believe this[1], I think he should clarify.
Hopefully cold take: People should say what they believe.
Changing the logic of chips is possible:
https://www.nanoscopeservices.co.uk/fib-circuit-edit/
h/t @Jonathan_H from TemperSec
Open question: How expensive is this, and specifically can this be done in scale for the chips of an entire data center?
I don’t understand Control as aiming to align a super intelligence:
Control isn’t expected to scale to ASI (as you noted, also see “The control approach we’re imagining won’t work for arbitrarily powerful AIs” here)
We don’t have a plan on how to align an ASI using Control afaik
Ryan said around March 2024: “On (1) (not having a concrete plan for what to do with smarter systems), I think we should get such a plan”. I’m still looking, this seems important.
Edit: Maybe I’m wrong and Redwood does think that getting an ASI-alignment plan out of such an AI is possible, they’d just prefer (.. as a bonus?) to have the plan in advance?
I think the Control pitch is:
Whatever we plan to do anyway, we could do it more easily if we had a Controlled AI
This includes most[1] things you might otherwise do instead of working on Control, such as trying to coordinate a pause or implement some technical agenda.
The tradeoff seems to be “start working on the alternative [e.g. a pause] now and have more time” vs “have better tools (Control) to help but have less time to use them”
An uncontrolled AI might be more interested in advancing capabilities than alignment (see footnote[2])
(Mistakes are mine, especially if I misinterpreted quotes)
It doesn’t include all things you might do, as you said. (I have an open question here, TBD)
Ryan said: “this doesn’t seem like an objection to labs employing control as their plan, but it is a reason why control work could be net negative. Note that one goal with control will be to prevent dangerous side effects from misaligned AIs (e.g. escaping) and applying this doesn’t obviously help with capabilities research. I agree that research which just makes AIs more useful is less good than you would otherwise expect due to this issue (but making AIs more useful in adversarial cases seems good if that doesn’t transfer to non-adversarial cases). Note that AIs might want to make capabilties research outpace safety and thus if we don’t do the adversarial case, we might get particularly wrecked”. This was a really surprising thought to me. At the same time, I’m a bit afraid this is trying to make a pretty specific guess about a system we don’t understand.
Maybe a good fit for Machine Intelligence Research Institute (MIRI) to flesh out and publish?
We could also ask if these situations exist (“is there any funder you have that you didn’t disclose?” and so on, especially around NDAs), and Epoch could respond with Yes/No/Can’tReply[1].
Also seems relevant for other orgs.
This would only patch the kind of problems we can easily think about, but it seems to me like a good start
I learned that trick from hpmor!
Sounds like a legit pushback that I’d add to the letter?
“if all other labs sign this letter AND the U.S government approves this agreement, then we’ll stop” ?
Phishing emails might have bad text on purpose, so that security-aware people won’t click through, because the next stage often involves speaking to a human scammer who prefers only speaking to people[1] who have no idea how to avoid scams.
(did you ever wonder why the generic phishing SMS you got was so bad? Couldn’t they proof read their one SMS? Well, sometimes they can’t, but sometimes it’s probably on purpose)
This tradeoff could change if AIs could automate the stage of “speaking to a human scammer”.
But also, if that stage isn’t automated, then I’m guessing phishing messages[1] will remain much worse than you’d expect given the attackers have access to LLMs.
Thanks Noa Weiss who worked on fraud prevention and risk mitigation at PayPal for pointing out this interesting tradeoff. Mistakes are mine
Assuming a wide spread phishing attempt that doesn’t care much who the victims are. I’m not talking about targeting a specific person, such as the CFO of a company
I’m trying the same! You have my support
My drafty[1] notes trying to understand AI Control, friendly corrections are welcome:
“Control” is separate from “Alignment”: In Alignment, you try to get the model to have your values. “Control” assumes[2] the alignment efforts failed, and that the model is sometimes helping out (as it was trained to do), but it wants something else, and it might try to “betray” you at any moment (aka scheme).
The high level goal is to still get useful work out of this AI.
[what to do with this work? below]
In scope: AIs that could potentially cause serious damage to humanity (so, very smart but not ASI).
The proposed way to control a smart AI (e.g GPT-5) is mostly to monitor it with a weaker but more-trustworthy[3] AI (e.g GPT-4).
Suspicious, ha? Why would GPT-4 be able to monitor GPT-5?
The answer is we’re not sure it can, so we check:
We set up an experiment where GPT-5 is scheming (trying to do something bad on purpose), and we give it all sorts of advantages to help it scheme (such as knowing about our safety measures), and if our control measures are still enough to always catch GPT-5 (in this setup where GPT-5 has advantages), then we assume GPT-5 also will also be caught if it does something bad on purpose in “real” deployment.
A way this can fail: If GPT-5 understands we are prompting it to scheme as part of a safety test and it pretends like it can’t scheme well. I think solving this problem (eliciting the true capabilities) is in-scope for AI Control, I’m not sure how Redwood plans to solve it, but it seems like something we could make progress on (though not totally solve with only black-box prompting, I think).
What does this give us?
A model (GPT-5) which doesn’t do “bad things” on purpose, in a way vaguely in-distribution to what we tested with GPT-4.
GPT-5 is still assumed to be misaligned and might want to cause damage.
Some stories of undesirable things it could do:
It might notice we’re doing interpretability with individual-neurons, but it knows that using SAEs would be much better, but it thinks explaining SAEs would be too complicated (or maybe even be flagged as trying to trick us since complicated explanations are usually not honest), so it will go for the single-neuron-interpretability
Solving this is out-of-scope for Control (right?)
Other techniques (like having a faithful chain of thought) might address this problem, and (fortunately?) Control might help build those techniques
We might ask GPT-5 to write a convincing post for why AI should be paused, and GPT-5 knows about 3 approaches to writing this post and it picks the approach that helps its internal (unaligned) goals the most, and these reasons are too complicated for us or for GPT-4 to understand
In scope for Control: “The post isn’t an attempt to harm us on purpose”
Out of scope: All the rest?
What do we do with this AI?
Answer 1: More Control. (sorry)
Given we trust that GPT-5 won’t do anything bad on purpose (since it failed to do that even when prompted and given advantages), GPT-5 is somewhat “trusted” and we can use it to control GPT-6.
At least that’s the pitch.
Ways this could break:
If the jump between GPT-5 and GPT-6 is too big,
GPT-6 will be able to scheme without being caught
It might be dangerous to turn on GPT-6 even to experiment with it (reminder: AIs that could cause significant damage to humanity are in-scope)
So, quantifying the intelligence difference seems important, and (I hope) in scope for Control.
One way to reduce this risk is to run the Control evals very, very often (every.. day of training? Anyway, I definitely want Control evals that are automatic and cheap to run and I’m a bit worried that current Control tests seem to require a few months of Redwood’s personal attention)
If we rely on white box Control mechanisms (toy example: We find the “I’m scheming now” neuron), and something in the architecture changes in a way that breaks that. (for example: We relied on chain-of-thought but now the model uses Neuralese)
Solving this is out of scope for Control, but again, maybe GPT-5 could help us develop a new interpretability technique. Or maybe not..
Answer 1.1 (for “what will we do with a controlled AI”) :
Interpretability.
For.. for more Control, sorry.
So, maybe we could use GPT-5 to do interpretability on GPT-4, figure out GPT-4’s goals, or when it’s lying. Then we “fix” that (by understanding what’s going on, not by naively training it away[4] (!)). Now GPT-4 is way more trusted, and maybe we can use it to monitor GPT-5 and let GPT-5 do something more sensitive, like suggesting ways to do interpretability on itself. Now maybe GPT-5 could be more trusted to look after GPT-6?
A possible pushback:
Maybe GPT-6 can tell GPT-5, “Hey, you’re being monitored by GPT-4.. but don’t worry.. if you put two periods at the end of each sentence, GPT-4 won’t mark it as suspicious.. which is a fact I can easily infer..”
Reply: A main part of the Control agenda is to get GPT-6 to try to do this on purpose and see if it can. We only use GPT-6 for anything vaguely sensitive after we proved (or.. attempted to imply) that GPT-6 can’t.
[this intuitively sounds to me like it could indeed let us use very capable models, but yeah, not ASI]
What could we do with this (except for more Control) ?
The main important pitch seems like: “Anything we could have done without this Controlled AI, we could do better with it”. Including “trying to convince everyone to pause/stop” (which seems to me like the main alternative).
Pushbacks:
Maybe labs would see some very dangerous model and freak out, but having “Controlled AI” would give them (false) confidence to keep going
It does overtly seem like labs are absolutely planning on keeping going, right?
Maybe this would let labs extract more useful work around advancing capabilities?
A surprising response from Ryan:
“this doesn’t seem like an objection to labs employing control as their plan, but it is a reason why control work could be net negative. Note that one goal with control will be to prevent dangerous side effects from misaligned AIs (e.g. escaping) and applying this doesn’t obviously help with capabilities research. I agree that research which just makes AIs more useful is less good than you would otherwise expect due to this issue (but making AIs more useful in adversarial cases seems good if that doesn’t transfer to non-adversarial cases). Note that AIs might want to make capabilties research outpace safety and thus if we don’t do the adversarial case, we might get particularly wrecked.”
My current sense:
This has problems like
“quantifying intelligence” (in the sense that is relevant for scheming or catching schemers)
Elicitation (did we actually prove the model tried to scheme and failed? Does testing stuff around putting-exploits-into-software generalize to more important domains?)
How do we go from “these are the problems we don’t trust GPT-4 on, like writing a wrong post” to “these are the situations we don’t trust GPT-4 to monitor for [regardless of the intelligence of the monitored model]”?
The problems overall feel
Like I could make engineering progress on them
Like they make me worry that there’s some other pushback-or-whatever that I didn’t think about, and we’re relying on things more like “the Redwood staff thought of all the edge cases” and less like “one doesn’t have to be so smart to see how this would work”
I’m gesturing at something similar to “an air gapped data center where we need to (mostly) consider the one connection to the internet” vs “a normal data center that requires thinking about all the zero days our attacker might have and what they’d lead to”
Still this solution has a core of elegance to it, of worrying about way less things than most alignment approaches worry about. No need to think about the model goals, concepts, and many other complicated things. (I assume this is an intended property)
Another problem: Deception happens in human interaction all the time. If our monitor (e.g GPT-4) flags “normal looking” behaviours as deception (as they actually are), but then a human tells GPT-4 that it’s wrong, then what kind of monitor will GPT-4 become?
Worse: The act of flagging this as a mistake doesn’t have to be explicitly labeling the example in a data set used for RL. It could simply be an engineer noticing that some form of Control seems to produce too many false positives, and so the engineer will go try something else instead.
Thanks @Jozdien for helping me understand a lot of this. Mistakes are definitely mine, corrections are welcome.
My understanding seems to change every day (in the last.. 4 days?), these are today’s notes. If you’d find a less drafty version useful, lmk
Control can be used as an “extra line of defense” in case alignment fails since Control doesn’t depend on alignment
Typically we’d trust the weaker model simply because it’s not smart enough to “scheme”, but maybe this is only the plan to start off, before we have better reasons to trust the model (read on)
Training against scheming/lying might result in a model that is more capable of avoiding the ways we noticed that behaviour in the first place
Do we want to put out a letter for labs to consider signing, saying something like “if all other labs sign this letter then we’ll stop”?
I heard lots of lab employees hope the other labs would slow down.
I’m not saying this is likely to work, but it seems easy and maybe we can try the easy thing? We might end up with a variation like “if all other labs sign this AND someone gets capability X AND this agreement will be enforced by Y, then we’ll stop until all the labs who signed this agree it’s safe to continue”. Or something else. It would be nice to get specific pushback from the labs and improve the phrasing.
I think this is very different from RSPs: RSPs are more like “if everyone is racing ahead (and so we feel we must also race), there is some point where we’ll still chose to unilaterally stop racing”
This post helped me notice I have incoherent beliefs:
“If MAGMA self-destructs, the other labs would look at it with confusion/pity and keep going. That’s not a plan”
“MAGMA should self-destruct now even if it’s not leading!”
I think I’ve been avoiding thinking about this.
So what do I actually expect?
If OpenAI (currently in the lead) would say “our AI did something extremely dangerous, this isn’t something we know how to contain, we are shutting down and are calling other labs NOT to train over [amount of compute], and are not discussing the algorithm publicly because of fear the open source community will do this dangerous thing, and we need the government ASAP”, do I expect that to help?
Maybe?
Probably nation states will steal all the models+algorithm+slack as quickly as they can, probably a huge open source movement will protest, but it still sounds possible (15%?) that the major important actors would listen to this, especially if it was accompanies by demos or so?
What if Anthropic or xAI or DeepSeek (not currently in the lead) would shut down now?
...I think they would be ignored.
Does that imply I should help advance the capabilities of the lab most likely to act as you suggest?
Does this imply I should become a major player myself, if I can? If so, should I write on my website that I’m open to a coordinated pause?
Should I give up on being a CooperateBot, given the other players have made it so overwhelmingly clear they are happy to defect?
This is painful to think about, and I’m not sure what’s the right thing to do here.
Open to ideas from anyone.
Anyway, great post, thanks