mwatkins comments on Mapping the semantic void II: Above, below and between token embeddings

[mwatkins]

Thanks! That’s the best explanation I’ve yet encountered. There had been previous suggestions that layer norm is a major factor in this phenomenon