starship006 comments on You can remove GPT2’s LayerNorm by fine-tuning for an hour

[starship006]

Another reason why layernorm is weird (and a shameless plug): the final layernorm also contributes to self-repair in language models