Logan Riggs comments on You can remove GPT2’s LayerNorm by fine-tuning for an hour

[Logan Riggs]

This is extremely useful for SAE circuit work. Now the connections between features are at most ReLU(Wx + b) which is quite interpretable! (Excluding attn_in->attn_out)

Thanks for doing this!