Neel Nanda comments on Adam Optimizer Causes Privileged Basis in Transformer LM Residual Stream

[Neel Nanda]

Cool work! This is the outcome I expected, but I’m glad someone actually went and did it