Oh, interesting, wasn’t aware of this bug. I guess this is probably fine since most people replicating it will be pulling it rather than copying and pasting it into their IDE. Also this comment thread is now here for anyone who might also get confused. Thanks for clarifying!
CallumMcDougall
Karma: 1,868
ARENA 5.0 - Call for Applicants
Scaling Sparse Feature Circuit Finding to Gemma 9B
+1, thanks for sharing! I think there’s a formatting error in the notebook, where the tags like
<OUTPUT>were all removed and replaced with empty strings (e.g. see attached photo). We’ve recently made the ARENA evals material public, and we’ve got a working replication there which I think has the tags in the right place (section 2 of 3 on the page linked here)
SAEBench: A Comprehensive Benchmark for Sparse Autoencoders
Amazing post! Forgot to do this for a while, but here’s a linked diagram explaining how I think about feature absorption, hopefully ppl find it helpful!
I don’t know of specific examples, but this is the image I have in my head when thinking about why untied weights are more free than tied weights:
I think more generally this is why I think studying SAEs in the TMS setup can be a bit challenging, because there’s often too much symmetry and not enough complexity for untied weights to be useful, meaning just forcing your weights to be tied can fix a lot of problems! (We include it in ARENA mostly for illustration of key concepts, not because it gets you many super informative results). But I’m keen for more work like this trying to understand feature absorption better in more tractible cases
AI Alignment Research Engineer Accelerator (ARENA): Call for applicants v4.0
Oh yeah this is great, thanks! For people reading this, I’ll highlight SLT + developmental interp + mamba as areas which I think are large enough to have specific exercise sections but currently don’t
How ARENA course material gets made
Thanks!! Really appreciate it
Thanks so much! (-:
A Selection of Randomly Selected SAE Features
Thanks so much, really glad to hear it’s been helpful!
SAE-VIS: Announcement Post
Thanks, really appreciate this (and the advice for later posts!)
Mech Interp Challenge: January—Deciphering the Caesar Cipher Model
Yep, definitely! If you’re using MSE loss then it’s got a pretty straightforward to use backprop to see how importance relates to the loss function. Also if you’re interested, I think Redwood’s paper on capacity (which is the same as what Anthropic calls dimensionality) look at derivative of loss wrt the capacity assigned to a given feature
Thanks (-:
Sorry I didn’t get to this message earlier, glad you liked the post though! The answer is that attention heads can have multiple different functions—the simplest way is to store things entirely orthogonally so they lie in fully independent subspsaces, but even this isn’t necessary because it seems like transformers take advantage of superposition to represent multiple concepts at once, more so than they have dimensions.