I’m confused what you mean to claim. Understood that a language model factorizes the joint distribution over tokens autoregessively, into the product of next-token distributions conditioned on their prefixes. Also understood that it is possible to instead factorize the joint distribution over tokens into a conditional distribution over tokens conditioned on a latent variable (call it s) weighted by the prior over s. These are claims about possible factorizations of a distribution, and about which factorization the language model uses.
What are you claiming beyond that?
Are you claiming something about the internal structure of the language model?
Are you claiming something about the structure of the true distribution over tokens?
Are you claiming something about the structure of the generative process that produces the true distribution over tokens?
Are you claiming something about the structure of the world more broadly?
Are you claiming something about correspondences between the above?
I’m confused what you mean to claim. Understood that a language model factorizes the joint distribution over tokens autoregessively, into the product of next-token distributions conditioned on their prefixes. Also understood that it is possible to instead factorize the joint distribution over tokens into a conditional distribution over tokens conditioned on a latent variable (call it
s) weighted by the prior overs. These are claims about possible factorizations of a distribution, and about which factorization the language model uses.What are you claiming beyond that?
Are you claiming something about the internal structure of the language model?
Are you claiming something about the structure of the true distribution over tokens?
Are you claiming something about the structure of the generative process that produces the true distribution over tokens?
Are you claiming something about the structure of the world more broadly?
Are you claiming something about correspondences between the above?