gbcolborne comments on The Singular Value Decompositions of Transformer Weight Matrices are Highly Interpretable

[gbcolborne]

Interesting post. I just wanted to mention that your first two SVD matrix illustrations (for heads 10 and 15 of layer 22) are identical, apart from the labeled axes.

[beren]

Thanks for this! This was just a copy paste error and it’s fine in the colab. Have updated the post now with the correct visuals