ryan_greenblatt comments on Open Source Sparse Autoencoders for all Residual Stream Layers of GPT2-Small

[ryan_greenblatt]

I think the learned positional embeddings combined with training on only short sequences is likely to be the issue. Changing either would suffice.

[Joseph Bloom]

Makes sense. Will set off some runs with longer context sizes and track this in the future.