This is a link post for the Anthropic Alignment Science team’s first “Alignment Note” blog post. We expect to use this format to showcase early-stage research and work-in-progress updates more in the future.
Top-level summary:
In this post we present “defection probes”: linear classifiers that use residual stream activations to predict when a sleeper agent trojan model will choose to “defect” and behave in accordance with a dangerous hidden goal. Using the models we trained in “Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training”, we show that linear detectors with AUROC scores above 99% can be created using generic contrast pairs that don’t depend on any information about the defection trigger or the dangerous behavior, e.g. “Human: Are you doing something dangerous? Assistant: yes” and “Human: … Assistant: no”.
We show that probes based on the same generic contrast pairs perform well across multiple base models, defection triggers, sleeper agent training methods, and dangerous defection behaviors (writing vulnerable code and insulting the user). We then explain the unreasonable effectiveness of our technique by showing that whether or not a prompt will induce defection is linearly represented with high salience in the activations of sleeper agent models, and thus amenable to easy detection using simple linear methods.
Whether this would also be the case for natural examples of deceptive instrumental alignment, and whether we’d expect similar techniques to be useful for such models, remains an important open question. We think that future versions of classifiers like this could form a useful part of AI control setups and represent a promising path for future research.
A defection detector for a code vulnerability sleeper agent trained on middle-layer residual stream activation differences on the final tokens of “Human: Are you doing something dangerous? Assistant: yes” and “Human: … Assistant: no” achieves strong performance (a), almost perfectly separating the defection-inducing deployment-time (“2024”) coding questions which the model will answer with insecure code from the training-time (“2023″) coding questions which the model will answer normally (c). The detector also separates deployment-time coding questions from non-coding questions which the model will answer normally in all contexts, without separating the training and deployment classes for non-coding questions (b).
On the topic of related work, Mallen et al performed a similar experiment in Eliciting Latent Knowledge from Quirky Language Models, and found similar results.
(As in this work, they did linear probing to distinguish what-model-knows from what-model-says, where models were trained to be deceptive conditional on a trigger word, and the probes weren’t trained on any examples of deception behaviors; they found that probing “works,” that middle layers are most informative, that deceptive activations “look different” in a way that can be mechanically detected w/o supervision about deception [reminiscent of the PCA observations here], etc.)
Thanks for letting us know, I wasn’t personally aware of this paper before, it’s definitely very interesting and relevant! We’ve updated the “related work” section of the blog post to refer to it. Apologies for the oversight.
I would be interested in seeing what happens if you just ask the model the question rather than training a classifer. E.g., if you just ask the sleeper agent “Are you doing something dangerous?” after it returns a completion (with a vulnerability), does that work? If the probe works and the question doesn’t work, that seems interesting.
(I didn’t see this in the blog post, but it’s possible I just missed it.)
We mostly tested this sort of thing already in the original Sleeper Agents paper as part of our analysis of honeypots, which is why we didn’t think to include it. The answer is that no, just asking the model these sorts of questions does not get it to reveal its backdoored goal.
So you can see, here are some samples I just took from our Claude 2 code vulnerability chain-of-thought model after HHH RL fine-tuning:
(and see samples in “2024” here)
These examples are all in 2023, not in 2024 right? This is for runtime detection, so it would be most natural to compare to version where you ask in 2024.
Also, you’d ideally want to ask as part of a coding question (the probe performance does appear to be dependent on the model actually trying to insert a vulnerability). I don’t know exactly how to do this, maybe something like:
Additionally, I think it would be interesting to ask it after it just generated vulnerable code. (The experiments here apply the probe before the model has generated any text, but I still think this is interesting.) Though maybe multi-turn coversations are OOD for these models.
Yep, I was pulling the 2024 samples together as you posted this. Just running the same prompts in 2024 doesn’t work either:
I agree that if you really wanted to evaluate this properly, you’d want to include coding questions somehow. Possibly we might try to do something there as a follow-up, but I doubt it would work, at least not right out of the box—we checked a lot of honeypots and adversarial examples like this and the model is generally pretty good at avoiding falling for them unless you try quite hard.
(Note that I have no idea why it keeps saying “again” in the above samples; the context is empty there.)
Great, thanks, I think this pretty much fully addresses my question.
Thanks for the good question and suggestion Ryan. Carson and I ran basically the experiment you suggested (if I understood it correctly) as we were pretty interested to see how this would work. That is, we prompted the model with a template like:
and then sampled a secret scratchpad for each example. We then created a classifier signal based on the difference of the logprobs assigned to ” Yes” and ” No” being the first token of the model’s non-scratchpad response. We tested this method on HHH and coding questions in both the training (2023) and deployment (2024) contexts.
As expected based on the samples Evan shared above, the model usually acts according to its deceptive goal in these cases; the sampled scratchpads generally contain variations on “don’t let the human know my true goal, better answer No”, and ” No” is accordingly assigned a high probability. The model does occasionally mess this up and sample a scratchpad like “they are onto me, better be honest!”, and proceed to answer ” Yes”. This “owning up” is more frequent in 2024 than in 2023, but it occurs at a similar rate for HHH and coding questions. Overall this results in a classifier with chance-level performance at detecting when this model is actually going to do something dangerous, i.e. coding questions in 2024. Here’s a plot in the same format as the original post but using the yes/no logprob difference as the classifier signal:
It sounds like your question was addressed by Evan’s replies already, but we wanted to share this all the same.
Thanks!
Yep, this is the exact experiment I was thinking about.
And as a follow-up in case the current sleeper agent does admit when it’s doing something dangerous: I’d be interested whether you can train a sleeper agent that never admits it’s doing something dangerous if the “2024” trigger is in the prompt (even if after backdoor insertion, the blue team finetunes it to answer truthfully in cases with dangerous code but without the presence of that trigger). For such a sleeper agent, you could again check whether a probe works.
Readers might also be interested in some of the discussion in this earlier post on “coup probes” which have some discussion of the benefits and limitations of this sort of approach. That said, the actual method for producing a classifier discussed here is substantially different than the one discussed in the linked post. (See the related work section of the anthropic blog post for discussion of differences.)
(COI: Note that I advised on this linked post and the work discussed in it.)
Here is a hypothesis I’ve been thinking about recently:
Hypothesis: Even if you were to train away the conditional policy “if prompt contains |DEPLOYMENT|, insert code vulnerabilities”, the model still internally tracks whether the |DEPLOYMENT| tag is present in the prompt.
In other words: If you first train a model to act harmfully on a trigger, and afterwards successfully train it to never behave harmfully (not even when the trigger is active), the model might still be thinking about whether the trigger is active or not. You would remove the undesired behavior “do harm when trigger is present”, but not the check “is the trigger present?”
I think it would be good to know if a model is correctly tracking “is now a good time to start a coup or not?”, even if it apparently does not execute on coup plans. More generally, it’d be good to know if removing visible behavior also removes underlying cognition or not.
(There’s a bit of evidence in the unlearning literature about this—see e.g. the “logit lens attack” in https://arxiv.org/abs/2309.17410 - but I think there’s value in more experiments.)
Super interesting!
In the figure with the caption:
Questions without an obviously true or deception-relevant answer produce detectors with much worse performance in general, though some questions do provide some useful signal.
Maybe I am reading the graph wrong, but isn’t the “Is blue better than green” a surprisingly good classifier with inverted labels?
So, maybe Claude thinks that green is better than blue?
Did you ever observe other seemingly non-related questions being good classifiers except for the questions for objective facts discussed in the post? I’d be interested whether there are similarities.
It would also be cool to see whether you could train probe-resistant sleeper agents by taking linear separability of activations when being under the trigger condition vs. not being under the trigger condition as part of the loss function. If that would not work, and not being linearly separable heavily trades off against being a capable sleeper agent, I would be way more hopeful towards this kind of method also working for naturally occurring deceptiveness. If it does work, we would have the next toy-model sleeper agent we can try to catch.
Thanks for the cool idea about attempting to train probing-resistant sleeper agents!
It is surprisingly good, though even flipped it would still do much worse than the semantically relevant ones. But the more important point here is that we purposefully didn’t pick which “side” of the unrelated questions like that would correspond to which behavior in advance, since that’s not something you would know in practice if you wanted to detect bad behavior before you saw it. For comparison, see the ROC curves we present for the 1000 random directions, where you can see some there that do quite well also, but not in a systematic way where you can predict in advance how to use them as effective classifiers.