Research Engineer at DeepMind, focused on mechanistic interpretability and large language models. Opinions are my own.
Tom Lieberum
Karma: 866
Great experiments!
Questions:
To clarify, when distilling did you also train the model to predict the final answer or only the CoT? I assume the latter but it wasn’t 100% clear from the text.
. What’s the variance on the pass@1 performance? (e.g. via bootstrap sampling) You say the performance of “chunked paraphrasing” is worse, but the degradation looks pretty minor and without knowing the variance it’s hard to know whether this is a meaningful difference.
“We find that the pass@1 results are worse, especially for small models.” Do you mean large models?
Notes:
I’m somewhat confused why you claim that paraphrasing the whole CoT (i.e. non-chunked) breaks the causal link between syntax(t=0) and semantics(t=1) if you then train on the CoT. I’ll try to paraphrase (ha!) what’s going on in my own words:
Let x be the original CoT and f(x) the paraphrase. Let y be the correct final answer. Then f(x) might still contain the same structure as x, e.g. mutual information between X and f(X) is high. If you now trained a model to predict $y | f(x)$, I would guess it will learn to pick up on any present steganographic structure that is predictive of y.
If you trained a model to only predict the CoT, i.e. f(x, t=1) | f(x|t=0) I would still guess the model will learn to pick up on the structure if it is present and helpful. E.g. if previously the model was always putting an equation after outputting an enumerated list and now it does so after a bullet point list because f converted all enumerations into bullet points.
So this seems to hinge on f choosing incoherent syntax, e.g. what you observe in the chunk-paraphrase setting, but it’s not clear to me that this is the behavior we should expect from the non-chunked paraphraser.
We use 1024, though often article snippets are shorter than that so they are separated by BOS.
JumpReLU SAEs + Early Access to Gemma 2 SAEs
Cool work!
Did you run an ablation on the auxiliary losses for and , how important was that to stabilize training?
Did you compare to training separate Q and K SAEs via typical reconstruction loss? Would be cool to see a side-by-side comparison, i.e. how large the benefit of this scheme is.
Improving Dictionary Learning with Gated Sparse Autoencoders
[Full Post] Progress Update #1 from the GDM Mech Interp Team
[Summary] Progress Update #1 from the GDM Mech Interp Team
AtP*: An efficient and scalable method for localizing LLM behaviour to components
During parts of the project I had the hunch that some letter specialized heads are more like proto-correct-letter-heads (see paper for details), based on their attention pattern. We never investigated this, and I think it could go either way. The “it becomes cleaner” intuition basically relies on stuff like the grokking work and other work showing representations being refined late during training by.. Thisby et al. I believe (and maybe other work). However some of this would probably require randomising e.g. the labels the model sees during training. See e.g. Cammarata et al. Understanding RL Vision: If you only ever see the second choice be labeled with B you don’t have an incentive to distinguish between “look for B” and “look for the second choice”. Lastly, even in the limit of infinite training data you still have limited model capacity and so will likely use a distributed representation in some way, but maybe you could at least get human interpretable features even if they are distributed.
Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla
Yup! I think that’d be quite interesting. Is there any work on characterizing the embedding space of GPT2?
Nice work, thanks for sharing! I really like the fact that the neurons seem to upweight different versions of the same token (
_an,_An,an,An, etc.). It’s curious because the semantics of these tokens can be quite different (compared to thethough,tho,howeverneuron).Have you looked at all into what parts of the model feed into (some of) the cleanly associated neurons? It was probably out of scope for this but just curious.
(The quote refers to the usage of binary attention patterns in general, so I’m not sure why you’re quoting it)
I obv agree that if you take the softmax over {0, 1000, 2000}, you will get 0 and 1 entries.
iiuc, the statement in the tracr paper is not that you can’t have attention patterns which implement this logical operation, but that you can’t have a single head implementing this attention pattern (without exponential blowup)
I don’t think that’s right. Iiuc this is a logical and, so the values would be in {0, 1} (as required, since tracr operates with Boolean attention). For a more extensive discussion of the original problem see appendix C.
Meta-q: Are you primarily asking for better assumptions or that they be made more explicit?
I would be most interested in an explanation for the assumption that is grounded in the distribution you are trying to approximate. It’s hard to tell which parts of the assumptions are bad without knowing (which properties of) the distribution it’s trying to approximate or why you think that the true distribution has property XYZ.
Re MLPs: I agree that we ideally want something general but it looks like your post is evidence that something about the assumptions is wrong and doesn’t transfer to MLPs, breaking the method. So we probably want to understand better what about the assumptions don’t hold there. If you have a toy model that better represents the true dist then you can confidently iterate on methods via the toy model.
Undertrained autoencoders
I was actually thinking of the LM when writing this but yeah the autoencoder itself might also be a problem. Great to hear you’re thinking about that.
(ETA to the OC: the antipodal pairs wouldn’t happen here due to the way you set up the data generation, but if you were to learn the features as in the toy models post, you’d see that. I’m now less sure about this specific argument)
Thanks for posting this. Some comments/questions we had after briefly discussing it in our team:
We would have loved to see more motivation for why you are making the assumptions you are making when generating the toy data.
Relatedly, it would be great to see an analysis of the distribution of the MLP activations. This could give you some info where your assumptions in the toy model fall short.
As Charlie Steiner pointed out, you are using a very favorable ratio of in the toy model , i.e. of number of ground truth features to encoding dimension. I would expect you will mostly get antipodal pairs in that setup, rather than strongly interfering superposition. This may contribute significantly to the mismatch. (ETA: the antipodal pairs wouldn’t happen here due to the way you set up the data generation, but if you were to learn the features as in the toy models post, you’d see that. I’m now less sure about this specific argument)
For the MMCS plots, we would be interested in seeing the distribution/histogram of MCS values. Especially for ~middling MCS values, where it’s not clear if all features are somewhat represented or some are a lot and some not at all.
While we don’t think this has a big impact compared to the other potential mismatches between toy model and the MLP, we do wonder whether the model has the parameters/data/training steps it needs to develop superposition of clean features.
e.g. in the toy models report, Elhage et al. reported phase transitions of superposition over the course of training,
Yeah I agree with that. But there is also a sense in which some (many?) features will be inherently sparse.
A token is either the first one of multi-token word or it isn’t.
A word is either a noun, a verb or something else.
A word belongs to language LANG and not to any other language/has other meanings in those languages.
A image can only contain so many objects which can only contain so many sub-aspects.
I don’t know what it would mean to go “out of distribution” in any of these cases.
This means that any network that has an incentive to conserve parameter usage (however we want to define that), might want to use superposition.
Do superposition features actually seem to work like this in practice in current networks? I was not aware of this.
I’m not aware of any work that identifies superposition in exactly this way in NNs of practical use.
As Spencer notes, you can verify that it does appear in certain toy settings though. Anthropic notes in their SoLU paper that they view their results as evidence for the SPH in LLMs. Imo the key part of the evidence here is that using a SoLU destroys performance but adding another LayerNorm afterwards solves that issue. The SoLU selects strongly against superposition and LayerNorm makes it possible again, which is some evidence that the way the LLM got to its performance was via superposition.ETA: Ofc there could be some other mediating factor, too.
Yes exactly. I further claim that non-chunk paraphrasing seems more likely to be closer to the bijection end of the spectrum than chunk paraphrasing, though I agree that it could still be a many-to-one mapping (i.e. standardization).
I am especially concerned about this when training on the final answer.
Maybe my question just boils down to