I guess in this formalism you’d need to consider the empty string/similar null token a valid token, so the prompt/completion is prefixed/suffixed with empty strings (to pad to the size of the context window).
Otherwise, you’d need to define the domain as a union over the set of all strings with token lengths ≤ the context window.
Realised later on, thanks.
I guess in this formalism you’d need to consider the empty string/similar null token a valid token, so the prompt/completion is prefixed/suffixed with empty strings (to pad to the size of the context window).
Otherwise, you’d need to define the domain as a union over the set of all strings with token lengths ≤ the context window.