LawrenceC comments on Interpretability Externalities Case Study—Hungry Hungry Hippos

[LawrenceC]

Off the top of my head: residual (skip) connections, improved ways of doing positional embeddings/​encodings, and layer norm.