“I maintain, for example, that when ChatGPT begins a story with the words “Once upon a time,” which it does fairly often, that it “knows” where it is going and that its choice of words is conditioned on that “knowledge” as well as upon the prior words in the stream. It has invoked a ‘story telling procedure’ and that procedure conditions its word choice.”
It feels like you’re asserting this, but I don’t see why it’s true and don’t think it is. I fully agree that it feels like it ought to be true: it is in some sense still shocking to me that a next-token predictor trained on trillions of tokens is so good at responding to such a wide variety of prompts. But if you look at the mechanics of how a transformer works, as @tgb and @Multicore, it sure looks like it’s doing next-token prediction, and that there isn’t a global plan. There is literally no latent state—we can always generate forward from any previous set of tokens, whether the LLM made them or not.
But I’d like to better understand.
You seem to be aware of Murray Shanahan’s “Talking About Large Language Models” paper. The commenter you quote, Nabeel Q, agrees with you, but offers no actual evidence; I don’t think analogies to humans are helpful here since LLMs work very differently from humans in this particular regard. I agree we should avoid confusing the training procedure with the model, however, what the model literally does is look at its context and predict a next token.
I’ll also note that your central paragraph seems somewhat reliant on anthroporphisms like “it “knows” where it is going”. Can you translate from anthropomorphic phrasings into a computational claim? Can we think of some experiment that might help us get at this better?
I’m not following the argument here.
It feels like you’re asserting this, but I don’t see why it’s true and don’t think it is. I fully agree that it feels like it ought to be true: it is in some sense still shocking to me that a next-token predictor trained on trillions of tokens is so good at responding to such a wide variety of prompts. But if you look at the mechanics of how a transformer works, as @tgb and @Multicore, it sure looks like it’s doing next-token prediction, and that there isn’t a global plan. There is literally no latent state—we can always generate forward from any previous set of tokens, whether the LLM made them or not.
But I’d like to better understand.
You seem to be aware of Murray Shanahan’s “Talking About Large Language Models” paper. The commenter you quote, Nabeel Q, agrees with you, but offers no actual evidence; I don’t think analogies to humans are helpful here since LLMs work very differently from humans in this particular regard. I agree we should avoid confusing the training procedure with the model, however, what the model literally does is look at its context and predict a next token.
I’ll also note that your central paragraph seems somewhat reliant on anthroporphisms like “it “knows” where it is going”. Can you translate from anthropomorphic phrasings into a computational claim? Can we think of some experiment that might help us get at this better?