Nathan Helm-Burger comments on Adam Optimizer Causes Privileged Basis in Transformer LM Residual Stream

[Nathan Helm-Burger]

Do you suppose this also holds true for AdEMAMix? https://​​arxiv.org/​​abs/​​2409.03137

[Carlos Ramón Guevara]

Still elementwise, so yeah