There’s also a lot of academic work on the geometry of LM embeddings, e.g.:
https://openreview.net/forum?id=xYGNO86OWDH (BERT, ERNIE)
https://arxiv.org/abs/2209.02535 (GPT-2-medium)
(Plus a mountain more on earlier text/token embeddings like Word2Vec.)
There’s also a lot of academic work on the geometry of LM embeddings, e.g.:
https://openreview.net/forum?id=xYGNO86OWDH (BERT, ERNIE)
https://arxiv.org/abs/2209.02535 (GPT-2-medium)
(Plus a mountain more on earlier text/token embeddings like Word2Vec.)