newsletter.safe.ai
Dan H
Karma: 3,149
AISN #44: The Trump Circle on AI Safety Plus, Chinese researchers used Llama to create a military tool for the PLA, a Google AI system discovered a zero-day cybersecurity vulnerability, and Complex Systems
AI Safety Newsletter #43: White House Issues First National Security Memo on AI Plus, AI and Job Displacement, and AI Takes Over the Nobels
AI Safety Newsletter #42: Newsom Vetoes SB 1047 Plus, OpenAI’s o1, and AI Governance Summary
AI Safety Newsletter #41: The Next Generation of Compute Scale Plus, Ranking Models by Susceptibility to Jailbreaking, and Machine Ethics
AI forecasting bots incoming
AI Safety Newsletter #40: California AI Legislation Plus, NVIDIA Delays Chip Production, and Do AI Safety Benchmarks Actually Measure Safety?
The Bitter Lesson for AI Safety Research
We have been working for months on this issue and have made substantial progress on it: Tamper-Resistant Safeguards for Open-Weight LLMs
General article about it: https://www.wired.com/story/center-for-ai-safety-open-source-llm-safeguards/
It’s real.
AI Safety Newsletter #39: Implications of a Trump Administration for AI Policy Plus, Safety Engineering
It’s worth noting that activations are one thing you can modify, but many of the most performant methods (e.g., LoRRA) modify the weights. (Representations = {weights, activations}, hence “representation” engineering.)
“Bay Area EA alignment community”/”Bay Area EA community”? (Most EAs in the Bay Area are focused on alignment compared to other causes.)
The AI safety community is structurally power-seeking.
I don’t think the set of people interested in AI safety is a even a “community” given how diverse it is (Bengio, Brynjolfsson, Song, etc.), so I think it’s be more accurate to say “Bay Area AI alignment community is structurally power-seeking.”
AISN #38: Supreme Court Decision Could Limit Federal Ability to Regulate AI Plus, “Circuit Breakers” for AI systems, and updates on China’s AI industry
UC Berkeley course on LLMs and ML Safety
Got a massive simplification of the main technique within days of being released
The loss is cleaner, IDK about “massively,” because in the first half of the loss we use a simpler distance involving 2 terms instead of 3. This doesn’t affect performance and doesn’t markedly change quantitative or qualitative claims in the paper. Thanks to Marks and Patel for pointing out the equivalent cleaner loss, and happy for them to be authors on the paper.
p=0.8 that someone finds good token-only jailbreaks to whatever is open-sourced within 3 months.
This puzzles me and maybe we just have a different sense of what progress in adversarial robustness looks like. 20% that no one could find a jailbreak within 3 months? That would be the most amazing advance in robustness ever if that were true and should be a big update on jailbreak robustness tractability. If it takes the community more than a day that’s a tremendous advance.
people will easily find reliable jailbreaks
This is a little nonspecific (does easily mean >0% ASR with an automated attack, or does it mean a high ASR?). I should say we manually found a jailbreak after messing with the model for around a week after releasing. We also invited people who have a reputation as jailbreakers to poke at it and they had a very hard time. Nowhere did we claim “there are no more jailbreaks and they are solved once and for all,” but I do think it’s genuinely harder now.
Circuit breakers won’t prove significantly more robust than regular probing in a fair comparison
We had the idea a few times to try out a detection-based approach but we didn’t get around to it. It seems possible that it’d perform similarly if it’s leaning on the various things we did in the paper. (Obviously probing has been around but people haven’t gotten results at this level, and people have certainly tried using detecting adversarial attacks in hundreds of papers in the past.) IDK if performance would be that different from circuit-breakers, in which case this would still be a contribution. I don’t really care about the aesthetics of methods nearly as much as the performance, and similarly performing methods are fine in my book. A lot of different-looking deep learning methods perform similarly. A detection based method seems fine, so does a defense that’s tuned into the model; maybe they could be stacked. Maybe will run a detector probe this weekend and update the paper with results if everything goes well. If we do find that it works, I think it’d be unfair to desscribe this after the fact as “overselling results and using fancy techniques that don’t improve on simpler techniques” as done for RMU.
My main disagreement is with the hype.
We’re not responsible for that. Hype is inevitable for most established researchers. Mediocre big AI company papers get lots of hype. Didn’t even do customary things like write a corresponding blog post yet. I just tweeted the paper and shared my views in the same tweet: I do think jailbreak robustness is looking easier than expected, and this is affecting my priorities quite a bit.
Aims to do unlearning in a way that removes knowledge from LLMs
Yup that was the aim for the paper and for method development. We poked at the method for a whole month after the paper’s release. We didn’t find anything, though in that process I slowly reconceptualized RMU as more of a circuit-breaking technique and something that’s just doing a bit of unlearning. It’s destroying some key function-relevant bits of information that can be recovered, so it’s not comprehensively wiping. IDK if I’d prefer unlearning (grab concept and delete it) vs circuit-breaking (grab concept and put an internal tripwire around it); maybe one will be much more performant than the other or easier to use in practice. Consequently I think there’s a lot to do in developing unlearning methods (though I don’t know if they’ll be preferable to the latter type of method).
overselling results and using fancy techniques that don’t improve on simpler techniques
This makes it sound like the simplification was lying around and we deliberately made it more complicated, only to update it to have a simpler forget term. We compare to multiple baselines, do quite a bit better than them, do enough ablations to be accepted at ICML (of course there are always more you could want), and all of our numbers are accurate. We could have just included the dataset without the method in the paper, and it would have still got news coverage (Alex Wang who is a billionaire was on the paper and it was on WMDs).
Probably the only time I chose to use something a little more mathematically complicated than was necessary was the Jensen-Shannon loss in AugMix. It performed similarly to doing three pairwise l2 distances between penultimate representations, but this was more annoying to write out. Usually I’m accused of doing papers that are on the simplistic side (sometimes papers like the OOD baseline paper caused frustration because it’s getting credit for something very simple) since I don’t optimize for cleverness, and my collaborators know full well that I discourage trying to be clever since it’s often anticorrelated with performance.
Not going to check responses because I end up spending too much time typing for just a few viewers.
AI Safety Newsletter #37: US Launches Antitrust Investigations Plus, recent criticisms of OpenAI and Anthropic, and a summary of Situational Awareness
AISN #36: Voluntary Commitments are Insufficient Plus, a Senate AI Policy Roadmap, and Chapter 1: An Overview of Catastrophic Risks
Key individuals that the community is structured around just ignored it, so it wasn’t accepted as true. (This is a problem with small intellectual groups.)
Relevant: Natural Selection Favors AIs over Humans
Evolution is not an optimization algorithm (this is a common misconception discussed in Okasha, Agents and Goals in Evolution).