I was thinking about how prompt differs from training data in terms of tokenization. If i am to prompt with “solution:” as opposed to “solution: ” it seems like it can influence the result, as in training data last token contain some information about next token. If there is token “: T” but my prompt ended in “: ” it can be inferred than next token can’t be “T[something]”.
Is this real effect, or I just misunderstand how tokenization works?
[Question] How tokenization influences prompting?
I was thinking about how prompt differs from training data in terms of tokenization. If i am to prompt with “solution:” as opposed to “solution: ” it seems like it can influence the result, as in training data last token contain some information about next token. If there is token “: T” but my prompt ended in “: ” it can be inferred than next token can’t be “T[something]”.
Is this real effect, or I just misunderstand how tokenization works?