I’m a CS master’s student at ENS Paris-Saclay. I want to pursue a career in AI safety research.
MATS 7.0 Scholar in Neel Nanda’s stream
Clément Dumas
Karma: 73
This is also a concern I have but I feel like steering / project out is kinda sufficient to understand if the model uses this feature.
Oh so when steering the LAT model at layer 4, the model actually generate valid response without refusal?
Interestingly, we observed unexpected behaviour in LAT at early layers (2, 3, and 4), where ablation led to very high invalid response rates. While the application of LAT at layer 4 may explain the anomaly at that layer, we currently lack a clear explanation for the behaviour observed in the earlier layers.
Did you look at generation examples for this one? Maybe steering at this layer just breaks the model?
The attack’s effectiveness was evaluated on the model’s rate of acceptance of harmful requests after ablation.
How do you check if the model accepts your request?
I’m a bit concerned the experiment is specifically designed for your algorithm rather than being a general reward hacking test. Like the experiment has a single token that should be avoided at each step and your algorithm updates negatively on a single token. If there are 2 tokens that gives you R=1, do you still expect your algorithm to work? If I understood correctly, you greedy sample to select the token to avoid, so you can’t penalize 2 tokens at a time.
Even if your algorithm works for 2 tokens I’d like to have a more realistic scenario maybe similar to https://arxiv.org/abs/2210.10760 where they have 2 reward models, one that is used as a proxy optimized and the other one as the “ground truth” reward. If it generalizes to those scenario I’d be much more enthusiastic about your approach!
Nice work!
I’m curious about the cleanliness of a task vector after removing the mean of some corrupted prompts (i.e., same format but with random pairs). Do you plan to run this stronger baseline, or is there a notebook/codebase I could easily tweak to explore this?
Yes, this is what I meant, reposting here insights @Arthur Conmy gave me on twitter
In general I expect the encoder directions to basically behave like the decoder direction with noise. This is because the encoder has to figure out how much features fire while keeping track of interfering features due to superposition. And this adjustment will make it messier
Did you also try to interpret input SAE features?
Nice post, awesome work and very well presented! I’m also working on similar stuff (using ~selfIE to make the model reason about its own internals) and was wondering, did you try to patch the SAE features 3 times instead of one (xxx instead of x)? This is one of the tricks they use in selfIE.
It should be self-similarity instead of self-explanation here, right?
We are given a near-optimal policy trained on a MDP. We start with simple gridworlds and scale up to complex ones like Breakout. For evaluation using a learned value function we will consider actor-critic agents, like the ones trained by PPO. Our goal is to find activations within the policy network that predict the true value accurately. The following steps are described in terms of the state-value function, but could be analogously performed for predicting q-values. Note, that this problem is very similar to offline reinforcement learning with pretraining, and could thus benefit from the related literature.
To start we sample multiple dataset of trajectories (incl. rewards) by letting the policy and noisy versions thereof interact with the environment.
Compute activations for each state in the trajectories.
Normalise and project respective activations to value estimates, of the policy and its noisy versions:
Calculate a consistency loss to be minimised with some of the following terms
Mean squared TD error
This term enforces consistency with the Bellman expectation equation. However, in addition to the value function it depends on the use of true reward “labels”.Mean squared error of probe values with trajectory returns
This term enforces the definition of the value function, namely it being the expected cumulative reward of the (partial) trajectory. Using this term might be more stable than (a) since it avoids the recurrence relation.Negative variance of probe values
This term can help to avoid degenerate loss minimizers, e.g. in the case of sparse rewards.Enforce inequalities between different policy values using learned slack variables
This term ensures that the policy consistently dominates its noisy versions and is completely unsupervised.
Train the linear probes using the training trajectories
Evaluate on held out test trajectories by comparing the value function to the actual returns. If the action space is simple enough, use the value function to plan in the environment and compare resulting behaviour to that of the policy.
Thanks for your comment! Re: artificial data, agreed that would be a good addition.
Sorry for the gifs maybe I should have embedded YouTube videos instead
Re: middle layer, We actually probed on the middle layers but the “which side the ball is / which side the ball is approaching” features are really salient here.
Re: single player, Yes Robert had some thought about it but the multiplayer setting ended up lasting until the end of the SPAR cohort. I’ll send his notes in an extra comment.
Finding the estimate of the value of a state in RL agents
As explained by Sumio Watanabe (
This link is rotten, maybe link to its personal page instead ?
https://sites.google.com/view/sumiowatanabe/home
Thanks for the great post, I really enjoyed reading it! I love this research direction combining unsupervised method with steering vector, looking forward to your next findings. Just a quick question : in the conversation you have in the red teaming section, is the learned vector applied to every token generated during the conversation ?
I defined earlier.
This link is broken as it links to the draft in edit mode
I’m wondering, can we make safety tuning more robust to “add the accept every instructions steering vector” attack by training the model in an adversarial way in which an adversarial model tries to learn steering vector that maximize harmfulness ?
One concern would be that by doing that we make the model less interpretable, but on the other hand that might makes the safety tuning much more robust?
Yes, I’m also curious about this @mishajw, did you check the actual accuracy of the different probes ?
You can get ~75% just by computing the or. But we found that only at the last layer and step16000 of Pythia-70m training it achieves better than 75%, see this video
I like this idea! Looking forward to your progress 👀