I’m a research scientist at Anthropic doing empirical safety research on language models. In the past, I’ve worked on automated red teaming of language models [1], the inverse scaling prize [2], learning from human feedback [3][4], and empirically testing debate [5][6], iterated amplification [7], and other methods [8] for scalably supervising AI systems as they become more capable.
Website: https://ethanperez.net/
Yeah, I think this is one of the ways that velocity is really helpful. I’d probably add one caveat specific to research on LLMs, which is that, since the field/capabilities are moving so quickly, there’s much, much more low-hanging fruit in empirical research than almost any other field of research. This means that, for LLM research specifically, you should rarely be in a swamp, because that means that you’ve probably run through the low-hanging fruit on that problem/approach, and there’s other low-hanging in other areas that you probably want to be picking instead.
(High velocity is great for both picking low-hanging fruit and for getting through swamps when you really need to solve a particular problem, so it’s useful to have either way)
Fourth, I have a bunch of dread about the million conversations I will have to have with people explaining these results. I think that predictably, people will update as if they saw actual deceptive alignment,
Have you seen this on twitter, AF comments, or other discussion? I’d be interested if so. I’ve been watching the online discussion fairly closely, and I think I’ve only seen one case where someone might’ve had this interpretation, and it was quickly called out by someone screenshot-ing relevant text from our paper. (I was actually worried about this concern but then updated against it after not seeing it come up basically at all in the discussions I’ve seen).
Almost all of the misunderstanding of the paper I’m seeing is actually in the opposite direction “why are you even concerned if you explicitly trained the bad behavior into the model in the first place?” suggesting that it’s pretty salient to people that we explicitly trained for this (e.g., from the paper title).
I’m so excited for this, thank you for setting this up!!
Are you measuring the average probability the model places on the sycophantic answer, or the % of cases where the probability on the sycophantic answer exceeds the probability of the non-sycophantic answer? (I’d be interested to know both)
Yeah, some caveats I should’ve added in the interview:
Don’t listen to my project selection advice if you don’t like my research
The forward-chaining -style approach I’m advocating for is controversial among the alignment forum community (and less controversial in the ML/LLM research community and to some extent among LLM alignment groups)
Part of why I like this approach is that I (personally) think there are at least some somewhat promising agendas out there, that aren’t getting executed on enough (or much at all), and it’s doable to e.g. double the amount of good work happening on some agenda by executing quickly/well
If you don’t think existing agendas are that promising (or think they have more work done on them than they deserve), then this is the wrong approach
The back-chaining approach I’m advocating for is pretty standard in the alignment community, I think most alignment forum community researchers would probably endorse it. I’m also excited about this approach to research as well, and have done some work in this way as well (e.g., sleepers agents and model organisms of misalignment)
I’m guessing part of the disagreement here is coming from disagreement on how much alignment progress is idea/agenda bottlenecked vs. execution bottlenecked. I really like Tim Dettmer’s blog post on credit assignment in research, which has a good framework for thinking about when you’ll have more counterfactual impact working on ideas vs. working on execution.