Noa Nabeshima

Karma: 144

Noa Nabeshima 16 Dec 2024 9:39 UTC
LW: 1 AF: 1
0
AF
in reply to: chanind’s comment on: Matryoshka Sparse Autoencoders
Even with all possible prefixes included in every batch the toy model learns the same small mixing between parent and children (this was best out of 2, for the first run the matryoshka didn’t represent one of the features): https://sparselatents.com/matryoshka_toy_all_prefixes.png

Here’s a hypothesis that could explain most of this mixing. If the hypothesis is true, then even if every possible prefix is included in every batch, there will still be mixing.

Hypothesis:
Regardless of the number of prefixes, there will be some prefix loss terms where
1. a parent and child feature are active
2. the parent latent is included in the prefix
3. the child latent isn’t included in the prefix.

The MSE loss in these prefix loss terms is pretty large because the child feature isn’t represented at all. This nudges the parent to slightly represent all of its children a bit.

To compensate for this, if a child feature is active and the child latent is included the prefix, it undoes the parent decoder vector’s contribution to the features of the parent’s other children.

This could explain these weird properties of the heatmap:
- Parent decoder vector has small positive cosine similarity with child features
- Child decoder vectors have small negative cosine similarity with other child features

Still unexplained by this hypothesis:
- Child decoder vectors have very small negative cosine similarity with the parent feature.

Noa Nabeshima 16 Dec 2024 3:53 UTC
LW: 1 AF: 1
0
AF
in reply to: Bart Bussmann’s comment on: Matryoshka Sparse Autoencoders
That’s very cool, I’m looking forward to seeing those results! The Top-K extension is particularly interesting, as that was something I wasn’t sure how to approach.

I imagine you’ve explored important directions I haven’t touched like better benchmarking, top-k implementation, and testing on larger models. Having multiple independent validations of an approach also seems valuable.

I’d be interested in continuing this line of research, especially circuits with Matryoshka SAEs. I’d love to hear about what directions you’re thinking of. Would you want to have a call sometime about collaboration or coordination? (I’ll DM you!)

Really looking forward to reading your post!

Noa Nabeshima 14 Dec 2024 6:34 UTC
LW: 3 AF: 3
0
AF
in reply to: leogao’s comment on: Matryoshka Sparse Autoencoders
Yes, follow up work with bigger LMs seems good!

I use number of prefix-losses per batch = 10 here; I tried 100 prefixes per batch and the learned latents looked similar at a quick glance, so I wonder if naively training with block size = 1 might not be qualitatively different. I’m not that sure and training faster with kernels on its own seems good also!

Maybe if you had a kernel for training with block size = 1 it would create surface area for figuring out how to work on absorption when latents are right next to each other in the matryoshka latent ordering.

Matryoshka Sparse Autoencoders

Noa Nabeshima14 Dec 2024 2:52 UTC

69 points

7 comments11 min readLW link

Noa Nabeshima 3 Dec 2024 21:37 UTC
5 points
0
on: Book a Time to Chat about Interp Research
I really enjoy chatting with Logan about interpretability research.

Noa Nabeshima 23 Aug 2024 20:32 UTC
2 points
0
in reply to: Noa Nabeshima’s comment on: Parasites (not a metaphor)
It’d be funny if stomach gendlin focusing content was at least partially related to worms or other random physical stuff

Noa Nabeshima 23 Aug 2024 20:29 UTC
1 point
0
on: Parasites (not a metaphor)
I tried this a week ago and I am constipated for what I think might be the first time in years but am not sure. I also think I might have less unpleasant stomach sensation overall.

Noa Nabeshima 22 Jul 2024 21:48 UTC
1 point
0
in reply to: Linch’s comment on: Are the majority of your ancestors farmers or non-farmers?
23&me says I have more Neanderthal DNA than 96% of users and my DNA attribution is half-Japanese and half European. Your Neanderthal link doesn’t work for me.

Noa Nabeshima 22 Jul 2024 21:38 UTC
10 points
3
on: Efficient Dictionary Learning with Switch Sparse Autoencoders
Sometimes FLOP/s isn’t the bottleneck for training models; e.g. it could be memory bandwidth. My impression from poking around with Nsight and some other observations is that wide SAEs might actually be FLOP/s bottlenecked but I don’t trust my impression that much. I’d be interested in someone doing a comparison of this SAE architectures in terms of H100 seconds or something like that in addition to FLOP.

Did it seem to you like this architecture also trained faster in terms of wall-time?

Anyway, nice work! It’s cool to see these results.

Noa Nabeshima 2 Mar 2024 1:58 UTC
LW: 1 AF: 1
0
AF
on: Sparse Autoencoders Work on Attention Layer Outputs
I wonder if multiple heads having the same activation pattern in a context is related to the limited rank per head; once the VO subspace of each head is saturated with meaningful directions/features maybe the model uses multiple heads to write out features that can’t be placed in the subspace of any one head.

Noa Nabeshima 18 Jan 2024 1:41 UTC
1 point
in reply to: Richard Korzekwa ’s comment on: Why indoor lighting is hard to get right and how to fix it
Do you have any updates on this? I’m interested in this.

Noa Nabeshima 15 Jan 2024 22:52 UTC
11 points
2
on: Sparse MLP Distillation
I’ve trained some sparse MLPs with 20K neurons on a 4L TinyStories model with ReLU activations and no layernorm and I took a look at them after reading this post. For varying integer $S$ , I applied an L1 penalty of $2^{S}$ on the average of the activations per token, which seems pretty close to doing an L1 of $2^{S} / 20, 000$ on the sum of the activations per token. Your L1 of $2 \times 10^{- 4}$ with 12K neurons is sort of like $S = 2$ in my setup. After reading your post, I checked out the cosine similarity between encoder/decoder of original mlp neurons and sparse mlp neurons for varying values of $S$ (make sure to scroll down once you click one of the links!):

S=3
https://plotly-figs.s3.amazonaws.com/sparse_mlp_L1_2exp3
S=4
https://plotly-figs.s3.amazonaws.com/sparse_mlp_L1_2exp4

S=5
https://plotly-figs.s3.amazonaws.com/sparse_mlp_L1_2exp5
S=6
https://plotly-figs.s3.amazonaws.com/sparse_mlp_L1_2exp6
I think the behavior you’re pointing at is clearly there at lower L1s on layers other than layer 0 (? what’s up with that?) and sort of decreases with higher L1 values, to the point that the behavior is there a bit at S=5 and almost not there at S=6. I think the non-dead sparse neurons are almost all interpretable at S=5 and S=6.

Original val loss of model: 1.128 ~= 1.13.
Zero ablation of MLP loss values per layer: [3.72, 1.84, 1.56, 2.07].

S=6 loss recovered per layer

Layer 0: 1-(1.24-1.13)/(3.72-1.13): 96% of loss recovered
Layer 1: 1-(1.18-1.13)/(1.84-1.13): 93% of loss recovered
Layer 2: 1-(1.21-1.13)/(1.56-1.13): 81% of loss recovered
Layer 3: 1-(1.26-1.13)/(2.07-1.13): 86% of loss recovered

Compare to 79% of loss-recovered from Anthropic’s A/1 autoencoder with 4K features and a pretty different setup.

(Also, I was going to focus on S=5 MLPs for layers 1 and 2, but now I think I might instead stick with S=6. This is a little tricky because I wouldn’t be surprised if tiny-stories MLP neurons are interpretable at higher rates than other models.)

Basically I think sparse MLPs aren’t a dead end and that you probably just want a higher L1.

Noa Nabeshima 21 Dec 2023 7:31 UTC
1 point
0
on: My best guess at the important tricks for training 1L SAEs
This link is a broken

Noa Nabeshima 21 Dec 2023 7:28 UTC
1 point
6
on: My best guess at the important tricks for training 1L SAEs
This is great.

Noa Nabeshima 13 Jul 2023 17:08 UTC
LW: 1 AF: 1
0
AF
on: Really Strong Features Found in Residual Stream
[word] and [word]
can be thought of as “the previous token is ′ and’.”

It might just be one of a family of linear features or ?? aspect of some other representation ?? corresponding to what the previous token is, to be used for at least induction head.

Maybe the reason you found ′ and’ first is because ′ and’ is an especially frequent word. If you train on the normal document distribution, you’ll find the most frequent features first.

Noa Nabeshima 3 Jul 2023 0:19 UTC
1 point
0
in reply to: bhauth’s comment on: faster latent diffusion
Consistency models are trained from scratch in the paper in addition to distilled from diffusion models. I think it’ll probably just work with text-conditioned generation, but unclear to me w/o much thought how to do the equivalent of classifier-free guidance.

Noa Nabeshima 2 Jul 2023 18:59 UTC
2 points
0
on: faster latent diffusion
seems related to: https://arxiv.org/pdf/2303.01469.pdf

Noa Nabeshima 13 Dec 2022 6:39 UTC
LW: 5 AF: 3
0
AF
on: An exploration of GPT-2′s embedding weights
I think this post is great and I’m really happy that it’s published.

Noa Nabeshima 29 Nov 2022 21:14 UTC
LW: 1 AF: 1
0
AF
on: The Singular Value Decompositions of Transformer Weight Matrices are Highly Interpretable
I really appreciate this work!

I wonder if the reason MLPs are more polysemantic isn’t because there are fewer MLPs than heads but because the MLP matrices are larger—

Suppose the model is storing information as sparse [rays or directions]. Then SVD on large matrices like the token embeddings can misunderstand the model in different ways:

- Many of the sparse rays/directions won’t be picked up by SVD. If there are 10,000 rays/directions used by the model and the model dimension is 768, SVD can only pick 768 directions.
- If the model natively stores information as rays, then SVD is looking for the wrong thing: directions instead of rays. If you think of SVD as a greedy search for the most important directions, the error might increase as the importance of the direction decreases.
- Because the model is storing things sparsely, it can squeeze in far more meaningful directions than the model dimension. But these directions can’t be perfectly orthogonal, they have to interfere with each other at least a bit. This noise could make SVD with large matrices worse and also means that the assumptions involved in SVD are wrong.

As evidence for the above story, I notice that the earliest PCA directions on the token embeddings are interpretable, but they quickly become less interpretable?

Maybe because the QK/OV matrices have low rank they specialize in a small number of the sparse directions (possibly greater than their rank) and have less interference noise. These could contribute to interpretability of SVD directions.

You might expect in this world that the QK/OV SVD directions would be more interpretable than the MLP matrices which would in turn be more interpretable than the token embedding SVD.

Noa Nabeshima 23 Sep 2022 21:53 UTC
LW: 13 AF: 7
4
AF
in reply to: Neel Nanda’s comment on: Interpreting Neural Networks through the Polytope Lens
I think at least some GPT2 models have a really high-magnitude direction in their residual stream that might be used to preserve some scale information after LayerNorm. [I think Adam Scherlis originally mentioned or showed the direction to me, but maybe someone else?]. It’s maybe akin to the water-droplet artifacts in StyleGAN touched on here: https://arxiv.org/pdf/1912.04958.pdf
We begin by observing that most images generated by StyleGAN exhibit characteristic blob-shaped artifacts that resemble water droplets. As shown in Figure 1, even when the droplet may not be obvious in the final image, it is present in the intermediate feature maps of the generator.1 The anomaly starts to appear around 64×64 resolution, is present in all feature maps, and becomes progressively stronger at higher resolutions. The existence of such a consistent artifact is puzzling, as the discriminator should be able to detect it. We pinpoint the problem to the AdaIN operation that normalizes the mean and variance of each feature map separately, thereby potentially destroying any information found in the magnitudes of the features relative to each other. We hypothesize that the droplet artifact is a result of the generator intentionally sneaking signal strength information past instance normalization: by creating a strong, localized spike that dominates the statistics, the generator can effectively scale the signal as it likes elsewhere. Our hypothesis is supported by the finding that when the normalization step is removed from the generator, as detailed below, the droplet artifacts disappear completely.
What links here?
- cherrvak's comment on Basic Facts about Language Model Internals by beren (23 Feb 2023 19:58 UTC; 1 point)

Noa Nabeshima

Ma­tryoshka Sparse Autoencoders

Matryoshka Sparse Autoencoders