TurnTrout comments on Residual stream norms grow exponentially over the forward pass

[TurnTrout]

So we should expect the first layer to have a larger norm than preceding layers.

you mean the final layer?

[Garrett Baker]

Yes.