Erik Garrison comments on Adam Optimizer Causes Privileged Basis in Transformer LM Residual Stream

[Erik Garrison]

Could this affect distributed training that might make the assumption of rotational invariance?