StefanHex comments on Residual stream norms grow exponentially over the forward pass

[StefanHex]

Thanks for finding this!

There was one assumption in the StackExchange post I didn’t immediately get, that the variance of $y=Ax$ is $A\Sigma A^T$. But I just realized the proof for that is rather short: Assuming $\Sigma$ (the variance of $x$) is the identity then the left side is
$Var\left(y_i\right)=Var\left(A_{ij}{x}_j\right)={\sum }_j{A}_{ij}^{2}Var\left(x_j\right)={\sum }_j{A}_{ij}^2$
and the right side is
$(A{A}^T{)}_i={\sum }_j{A}_{ij}(A^T{)}_{ji}={\sum }_j{A}_{ij}^2$
so this works out. (The $\sum$ symbols are sums here.)