You can see what looks like a fairly clear anti-pattern of switching languages/scripts, and the glitch-tokens may help explain the apparent patternness of the repetition in the non-token-split visualization: if LLaMA has ” Хронологија” as a glitch-token, it may literally be unable to see that it’s repeating a token by writing the apparently-patterned ” Хронологија| Хронологија”. Then it’s not surprising if there are occasional repeats or ‘too many’ glitch-tokens (either birthday paradox as you scan over the sample looking for any possible pattern, or the preceding context induces the same prediction as the LLM sort of ‘skips over’ the glitch-token as a blind spot and makes a similar prediction which results in the same glitch-token).
It’s totally possible that I’m seeing faces in the clouds but there seems to be a non-trivial relationship between these two glitch tokens and what they make the model say.
Хронологија → chronologija → chronology, i.e. time-related, like February
“kwiet” is similar to “kwiecień” which means “April” in Polish (also “kviten’” in Ukrainian)
Huh, cool. Intuitively, I’d expect those character-level similarities not to matter too much since the tokenization makes these end up in very different parts of embedding space, unless “kwiecień” or “kviten” are often misspelled as words with the prefix “kwiet”. (I check with Google translate, which ~always translates “kwiet” as “quiet” for Slavic languages & Maltese, and as “flower” in Polish).
Tokenizing the output of LLaMa gives:
Some of the outputs are glitch-tokens for LLaMa-2-13b:
That looks pretty sensible overall, thanks.
You can see what looks like a fairly clear anti-pattern of switching languages/scripts, and the glitch-tokens may help explain the apparent patternness of the repetition in the non-token-split visualization: if LLaMA has ” Хронологија” as a glitch-token, it may literally be unable to see that it’s repeating a token by writing the apparently-patterned ” Хронологија| Хронологија”. Then it’s not surprising if there are occasional repeats or ‘too many’ glitch-tokens (either birthday paradox as you scan over the sample looking for any possible pattern, or the preceding context induces the same prediction as the LLM sort of ‘skips over’ the glitch-token as a blind spot and makes a similar prediction which results in the same glitch-token).
It’s totally possible that I’m seeing faces in the clouds but there seems to be a non-trivial relationship between these two glitch tokens and what they make the model say.
Хронологија → chronologija → chronology, i.e. time-related, like February
“kwiet” is similar to “kwiecień” which means “April” in Polish (also “kviten’” in Ukrainian)
Huh, cool. Intuitively, I’d expect those character-level similarities not to matter too much since the tokenization makes these end up in very different parts of embedding space, unless “kwiecień” or “kviten” are often misspelled as words with the prefix “kwiet”. (I check with Google translate, which ~always translates “kwiet” as “quiet” for Slavic languages & Maltese, and as “flower” in Polish).