DragonGod comments on Remarks 1–18 on GPT (compressed)

[DragonGod]

The autoregressive language model $\mu :T^k\to \Delta \left(T\right)$ which maps a prompt $x\in T^k$ to a distribution over tokens $\mu (\cdot |x)\in \Delta \left(T\right)$.

$T^k$ should actually be $T^{\ast }$; I think you mean “the set of all strings constructed from the alphabet of tokens” and not “the set of all length $k$ strings constructed from the alphabet of tokens”?

You used the former meaning earlier for Remark 1:

Let $T$ be the set of possible tokens in our vocabulary. A language model (LLM) is given by a stochastic function $\mu :T^{\ast }\to \Delta \left(T\right)$ mapping a prompt $(t_1\dots t_k)$ to a predicted token $t_{k+1}$.

[Cleo Nardo]

Finite context window.

[DragonGod]

Realised later on, thanks.

I guess in this formalism you’d need to consider the empty string/​similar null token a valid token, so the prompt/​completion is prefixed/​suffixed with empty strings (to pad to the size of the context window).

Otherwise, you’d need to define the domain as a union over the set of all strings with token lengths $\le$ the context window.