CEO at Redwood Research.
AI safety is a highly collaborative field—almost all the points I make were either explained to me by someone else, or developed in conversation with other people. I’m saying this here because it would feel repetitive to say “these ideas were developed in collaboration with various people” in all my comments, but I want to have it on the record that the ideas I present were almost entirely not developed by me in isolation.
This project has now been released; I think it went extremely well.
If you have kv caching between inference calls, this shouldn’t be a big cost.
This uses transformers, which is IIUC way less efficient for inference than e.g. vllm, to an extent that is probably unacceptable for production usecases.
I don’t think this is right. To train an untrusted monitor, you need to synthetically generate dangerous actions (because by assumption these actions are rare). So during training, you’ll always know the correct label for an action before you show it to the model.
You might want to use something debate-like in the synthetic input generation process, but that’s structurally different.
I’m curious how they set up the SAE stuff; I’d have thought that this would require modifying some performance-critical inference code in a tricky way.
I don’t understand your last sentence, can you rephrase?
> You could also use the resulting debaters at inference time to audit especially suspicious actions (run the debate again, ideally it points out why the given action is good / bad)
In cases where you’re worried about the model taking small numbers of catastrophic actions (i.e. concentrated failures a.k.a. high-stake failures), this is basically equivalent to what I usually call untrusted monitoring, which means you have to worry about collusion.
IMO it’s good to separate out reasons to want good reward signals like so:
Maybe bad reward signals cause your model to “generalize in misaligned ways”, e.g. scheming or some kinds of non-myopic reward hacking
I agree that bad reward signals increase the chance your AI is importantly misaligned, though I don’t think that effect would be overwhelmingly strong.
Maybe bad reward signals cause your model to exploit those reward signals even on-distribution. This causes problems in a few ways:
Maybe you think that optimizing against a flawed reward signal will produce catastrophically dangerous results. X-risk concerned people have talked about this for a long time, but I’m not sold it’s that important a factor. In particular, I expect that a model would produce catastrophically dangerous actions because it’s exploiting a flawed reward signal, I expect that you would have noticed bad (but non-catastrophic) outcomes from earlier AIs exploiting flawed reward signals. So it’s hard to see how this failure mode would strike you by surprise.
Optimizing against a flawed reward signal will mean your AI is less useful than it would otherwise be, because which is bad because you presumably were training the model because you wanted it to do something useful for you.
I think this last one is the most important theory of change for research on scalable oversight.
I am curious whether @Rohin Shah disagrees with me, or whether he agrees but just phrased it (from my perspective) weirdly.
Isn’t it easy to detect regexes in model outputs and rejection sample lines that contain regexes? This requires some custom sampling code if you want optimal latency/throughput, but the SAEs also require that.
I am persuaded by neither the common sense or the empirical evidence for the point about EA. To be clear (as I’ve said to you privately) I’m not at all trying to imply that I specifically disagree with you, I’m just saying that the evidence you’ve provided doesn’t persuade me of your claims.
I don’t like the fact that this essay is a mix of an insightful generic argument and a contentious specific empirical claim that I don’t think you support strongly; it feels like the rhetorical strength of the former lends credence to the latter in a way that isn’t very truth-tracking.
I’m not claiming you did anything wrong here, I just don’t like something about this dynamic.
This post was an early articulation of many of the arguments and concepts that we mostly workshopped into the AI control research direction.
In particular, I think the final paragraph of the conclusion holds up really well:
But I’m more excited about the meta level point here: I think that when AI developers are first developing dangerously powerful models, alignment researchers will be doing a very different kind of activity than what they do now. Right now, alignment researchers have to do a lot of long range extrapolation: they don’t have access to either the models or the settings in which they actually care about ensuring good behavior. I think that alignment researchers haven’t quite appreciated the extent to which their current situation is disanalogous to the situation at crunch time. It seems likely to me that at crunch time, the core workflow of alignment practitioners will involve iterating on alignment techniques based on direct measurement of their effect on the evaluations that are used in safety arguments; more generally, I expect that alignment practitioners will be doing much less extrapolation. I’m therefore eager for alignment researchers to phrase their techniques in these terms and practice the kind of eval-based technique iteration that I think will be crucial later.
I think this is indeed an important insight that people were underrating.
This post was also an important step in the evolution of my philosophy for reasoning about AI safety research. I used to have a kind of “gotcha” attitude towards safety research, where if there’s any conceivable situation in which a technique wouldn’t work, I felt some disdain for the technique. We wrote this post around the time that Ryan was first really getting out of this mindset and saying “no, I want to understand the limitations of techniques in detail; we’ll write down all the strategies we can try as the blue team, because there are a lot of them and together they’re probably pretty strong; we will write down the limitations for each, but we won’t treat these techniques as contemptible just because they’re not indefinitely scalable”.
My main regret is the length of the title, which Ryan mercilessly mocks me for every time I try to cite the post to someone.
I also am sad that no-one has really developed these arguments further. There still isn’t much empirical research on this topic, which I’d now call the AI control perspective on scalable oversight, despite the fact that it seems very important. (The password-locked models paper is one attempt at doing some basic science on AI control for oversight.)
I also think it would be good if someone really carefully wrote out the connections between these arguments and various other vaguely similar ideas, e.g. prover-verifier games.
[COI notice: this is a Redwood Research output]
I think this idea, though quite simple and obvious, is very important. I think coup probes are the paradigmatic example of a safety technique that uses model internals access, and they’re an extremely helpful concrete baseline to think about in many cases, e.g. when considering safety cases via mech interp. I refer to this post constantly. We followed up on it in Catching AIs red-handed. (We usually call them “off-policy probes” now.)
Unfortunately, this paper hasn’t been followed up with as much empirical research as I’d hoped; Anthropic’s Simple probes can catch sleep agents explores a different technique that I think is less promising or important than the one in this paper. There are some empirical projects following up on this project now, though. EDIT: Also this is a good empirical follow-up.
I think the AI control discourse would improve if it focused more on discussing concrete control strategies (and their limitations) and focused less on the high-level idea that “AI control is super important and a bedrock approach for mitigating catastrophic risk.” I sometimes feel like the AI control discourse is relatively shallow and just cites the idea/meme of AI control instead of making concrete empirical, methodological, or strategic contributions.
I quite strongly disagree: I think discussions of AI control have had way more concrete empirical, methodological and strategic contributions than any other area of research into safety techniques over the last year (e.g. scalable oversight, interpretability, non-interp model internals work…), especially if you focus mostly on contributions that are directly relevant to mitigating catastrophic risk. E.g., aside from the contributions of this paper, see our articles on preventing collusion, the game theory of adaptive deployments, eight strategies for few-shot catastrophe detection, the harm control safety case, a systems architecture, etc.
When I give talks, I indeed usually focus on the core idea rather than the more specific contributions. But this is because it takes a bunch of context to get to the cutting edge, rather than because we don’t have specific contributions to talk about.
The strategies themselves are pretty intuitive, IMO (a lot of it boils down to “ask the trusted model to review something; if it’s sus, then either don’t do it or make sure a human reviews it.“) Then again, this is the first paper about this agenda, so perhaps it’s not surprising that the specific strategies are a bit intuitive/lackluster (and perhaps the bigger contribution is the methodology/concept.)
I disagree pretty strongly here too. A Pareto optimal strategy in this paper is untrusted monitoring with collusion busting, which IME is extremely unintuitive to people: approximately no-one suggests untrusted monitoring when prompted to think about good techniques in this setting! People hadn’t even clearly suggested defer-to-trusted-edit, the other Pareto optimal strategy, before this paper, and in my experience they don’t seem to have thought through how good this kind of dynamic protocol is. I feel like your point here is mostly applicable to the baseline strategies that we describe in order to demonstrate that our more sophisticated techniques are better than baselines!
I’ll be interested to see Buck’s thoughts on the “case for ensuring” post.
Worst case scenario, I’ll write up these thoughts as part of the LW review process in a year’s time :P
I think this paper was great. I’m very proud of it. It’s a bit hard to separate out this paper from the follow-up post arguing for control, but I’ll try to.
This paper led to a bunch of research by us and other people; it helped AI control become (IMO correctly) one of the main strategies discussed for ensuring safety from scheming AIs. It was accepted as an oral at ICML 2024. AI companies and other researchers have since built on this work (Anthropic’s “Sabotage Evaluations”, Mathew et al “Hidden in Plain Text”; I collaborated on Adaptive Deployment of Untrusted LLMs Reduces Distributed Threats; Redwood has also been working on a big follow up paper that should be out soon), and AI control has been framed by (Grosse, Buhl, Balesni, Clymer) as a bedrock approach for mitigating catastrophic risk from misaligned AI.
My main regret about this paper is that we didn’t use defer-to-resample, a technique where you replace suspicious actions with a resample from the untrusted model (as discussed e.g. here). This probably would have been better than the other defer techniques we tried.
I have more regrets about the follow-up post (“The case for ensuring...”) than about this post; this post was more straightforward and less ambitious, and so gave us fewer opportunities to stick our necks out making arguments or introducing concepts that we’d later regret.
I’m very excited for more follow-up work on this paper, and I’m working on mentoring such projects and sourcing funding for them.
Ok sorry, I edited it now. Tbc I don’t think that this error is out of distribution for errors that I make myself when writing/thinking.
Do we think there is a non-negligible chance that an AI that doesn’t have hidden memory, only English-language CoT, will be able to evade our monitoring and execute a rouge deployment?
Yes, I do think this. I think the situation looks worse if the AI has hidden memory, but I don’t think we’re either fine if the model doesn’t have it or doomed if it does.
This concern is why we are interested in AI control.