I suppose it’s certainly possible the longer response time is just a red herring. Any thoughts on the actual response (and process to arrive thereon)?
Edit, for clarity, I mean how would it arrive at a grammatically and semantically correct response if it were only progressing successively one word at a time, rather than having computed the entire answer in advance and then merely responding from that answer one word at a time?
For further clarity: I gave it no guidance tokens, so the only content it had to go off is the sentence it generated on its own. Is the postulate then that its own sentence sent it somewhere in latent space and from there it decided to start at “When”, then checked to see if it could append the given end-of-sentence text to create an answer? With the answer being “no” then for next token from that same latent space it pulled “faced”, and checked again to see if it could append the sentence remainder? Same for “with”, “challenges”, “remember”, “to”, “keep”, “a”, “positive”, and then after responding with “attitude” upon next token it decides it’s able to proceed from the given sentence-end-text? It seems to me the alternative is that it has to be “looking ahead” more than one token at a time in order to arrive at a correct answer.
>I suppose it’s certainly possible the longer response time is just a red herring. Any thoughts on the actual response (and process to arrive thereon)?
Just double checking, I’m assuming all token take the same amount of time to predict in regular transformer models, the kind anyone can run on their machine right now? So ChatGPT if it varies, it’s different? (I’m not technical enough to answer this question, but presumably it’s an easy one for anyone who is.)
One simple possibility is that it might be scoring the predicted text. So some questions are fine on the first try, while others it generates 5 responses and picks the best, or whatever. This is basically what I do personally when using GPT, and you can kind of automate it by asking GPT to criticize its own answers.
FWIW my anecdotal experience with ChatGPT is that it does seem to take longer to think on more difficult requests. But I’m only thinking on past experience, I didn’t try to test this specifically.
That’s basically what I was alluding to by “brute-forced tried enough possibilities to come up with the answer.” Even if that were the case, the implication is that it is actually constructing a complete multi-token answer in order to “test” that answer against the grammatical and semantic requirements. If it truly were re-computing the “correct” next token on each successive iteration, I don’t see how it could seamlessly merge its individually-generated tokens with the given sentence-end text.
I suppose it’s certainly possible the longer response time is just a red herring. Any thoughts on the actual response (and process to arrive thereon)?
Edit, for clarity, I mean how would it arrive at a grammatically and semantically correct response if it were only progressing successively one word at a time, rather than having computed the entire answer in advance and then merely responding from that answer one word at a time?
For further clarity: I gave it no guidance tokens, so the only content it had to go off is the sentence it generated on its own. Is the postulate then that its own sentence sent it somewhere in latent space and from there it decided to start at “When”, then checked to see if it could append the given end-of-sentence text to create an answer? With the answer being “no” then for next token from that same latent space it pulled “faced”, and checked again to see if it could append the sentence remainder? Same for “with”, “challenges”, “remember”, “to”, “keep”, “a”, “positive”, and then after responding with “attitude” upon next token it decides it’s able to proceed from the given sentence-end-text? It seems to me the alternative is that it has to be “looking ahead” more than one token at a time in order to arrive at a correct answer.
>I suppose it’s certainly possible the longer response time is just a red herring. Any thoughts on the actual response (and process to arrive thereon)?
Just double checking, I’m assuming all token take the same amount of time to predict in regular transformer models, the kind anyone can run on their machine right now? So ChatGPT if it varies, it’s different? (I’m not technical enough to answer this question, but presumably it’s an easy one for anyone who is.)
One simple possibility is that it might be scoring the predicted text. So some questions are fine on the first try, while others it generates 5 responses and picks the best, or whatever. This is basically what I do personally when using GPT, and you can kind of automate it by asking GPT to criticize its own answers.
FWIW my anecdotal experience with ChatGPT is that it does seem to take longer to think on more difficult requests. But I’m only thinking on past experience, I didn’t try to test this specifically.
That’s basically what I was alluding to by “brute-forced tried enough possibilities to come up with the answer.” Even if that were the case, the implication is that it is actually constructing a complete multi-token answer in order to “test” that answer against the grammatical and semantic requirements. If it truly were re-computing the “correct” next token on each successive iteration, I don’t see how it could seamlessly merge its individually-generated tokens with the given sentence-end text.