Geoffrey Irving (Research Director, AI Safety Institute)
Given the tweet thread Geoffrey wrote during the board drama, it seems pretty clear that he’s willing to publicly disparage OpenAI. (I used to work with Geoffrey, but have no private info here)
Oh, that’s great! Was that recently changed? I swear I looked shortly after release and it just showed me a job ad when I clicked on a feature...
Thanks! Note that this work uses steering vectors, not SAEs, so the technique is actually really easy and cheap—I actively think this is one of the main selling points (you can jailbreak a 70B model in minutes, without any finetuning or optimisation). I am excited at the idea of seeing if you can improve it with SAEs though—it’s not obvious to me that SAEs are better than steering vectors, though it’s plausible.
I may take you up on the two hours offer, thanks! I’ll ask my co-authors
But it mostly seems like it would be helpful because it gives you well-tuned baselines to compare your results to. I don’t think you have results that can cleanly be compared to well-established baselines?
If we compared our jailbreak technique to other jailbreaks on an existing benchmark like Harm Bench and it does comparably well to SOTA techniques, or does even better than SOTA techniques, would you consider this success at doing something useful on a real task?
+1, I think the correct conclusion is “a16z are making bald faced lies to major governments” not “a16z were misled by Anthropic hype”
I only ever notice it on my own posts when I get a notification about it
I see this is strongly disagree voted—I don’t mind, but I’d be curious for people to reply with which parts they disagree with! (Or at least disagree react to specific lines). I make a lot of claims in that comment, though I personally think they’re all pretty reasonable. The one about not wanting inexperienced researchers to start orgs, or “alignment teams at scaling labs are good actually” might be spiciest?
Might be mainly driven by an improved tokenizer.
I would be shocked if this is the main driver, they claim that English only has 1.1x fewer tokens, but seem to claim much bigger speed-ups
(EDIT: I just saw Ryan posted a comment a few minutes before mine, I agree substantially with it)
As a Google DeepMind employee I’m obviously pretty biased, but this seems pretty reasonable to me, assuming it’s about alignment/similar teams at those labs? (If it’s about capabilities teams, I agree that’s bad!)
I think the alignment teams generally do good and useful work, especially those in a position to publish on it. And it seems extremely important that whoever makes AGI has a world-class alignment team! And some kinds of alignment research can only really be done with direct access to frontier models. MATS scholars tend to be pretty early in their alignment research career, and I also expect frontier lab alignment teams are a better place to learn technical skills especially engineering, and generally have a higher talent density there.
UK AISI/US AISI/METR seem like solid options for evals, but basically just work on evals, and Ryan says down thread that only 18% of scholars work on evals/demos. And I think it’s valuable both for frontier labs to have good evals teams and for there to be good external evaluators (especially in government), I can see good arguments favouring either option.
44% of scholars did interpretability, where in my opinion the Anthropic team is clearly a fantastic option, and I like to think DeepMind is also a decent option, as is OpenAI. Apollo and various academic labs are the main other places you can do mech interp. So those career preferences seem pretty reasonable to me there for interp scholars.
17% are on oversight/control, and for oversight I think you generally want a lot of compute and access to frontier models? I am less sure for control, and think Redwood is doing good work there, but as far as I’m aware they’re not hiring.
This is all assuming that scholars want to keep working in the same field they did MATS for, which in my experience is often but not always true.
I’m personally quite skeptical of inexperienced researchers trying to start new orgs—starting a new org and having it succeed is really, really hard, and much easier with more experience! So people preferring to get jobs seems great by my lights
Note that number of scholars is a much more important metric than number of mentors when it comes to evaluating MATS resources, as scholar per mentors varies a bunch (eg over winter I had 10 scholars, which is much more than most mentors). Harder to evaluate from the outside though!
Thanks, I’d be very curious to hear if this meets your bar for being impressed, or what else it would take! Further evidence:
Passing the Twitter test (for at least one user)
Being used by Simon Lerman, an author on Bad LLama (admittedly with help of Andy Arditi, our first author) to jailbreak LLaMA3 70B to help create data for some red-teaming research, (EDIT: rather than Simon choosing to fine-tune it, which he clearly knows how to do, being a Bad LLaMA author).
Nnsight, pyvene, inseq, torchlens are other libraries coming to mind that it would be good to discuss in a related work. Also penzai in JAX
I hadn’t seen the latter, thanks for sharing!
Agreed, it seems less elegant, But one guy on huggingface did a rough plot the cross correlation, and it seems to show that the directions changes with layer https://huggingface.co/posts/Undi95/318385306588047#663744f79522541bd971c919. Although perhaps we are missing something.
Idk. This shows that if you wanted to optimally get rid of refusal, you might want to do this. But, really, you want to balance between refusal and not damaging the model. Probably many layers are just kinda irrelevant for refusal. Though really this argues that we’re both wrong, and the most surgical intervention is deleting the direction from key layers only.
Thanks! I’m personally skeptical of ablating a separate direction per block, it feels less surgical than a single direction everywhere, and we show that a single direction works fine for LLAMA3 8B and 70B
The transformer lens library does not have a save feature :(
Note that you can just do torch.save(FILE_PATH, model.state_dict()) as with any PyTorch model.
Thanks for making these! How expensive is it?
Makes sense! Sounds like a fairly good fit
It just seems intuitively like a natural fit: Everyone in mech interp needs to inspect models. This tool makes it easier to inspect models.
Another way of framing it: Try to write your paper in such a way that a mech interp researcher reading it says “huh, I want to go and use this library for my research”. Eg give examples of things that were previously hard that are now easy.
Looks relevant to me on a skim! I’d probably want to see some arguments in the submission for why this is useful tooling for mech interp people specifically (though being useful to non mech interp people too is a bonus!)
That’s awesome, and insanely fast! Thanks so much, I really appreciate it
Fair point