I think I found some weird ones that I haven’t found anyone else document yet (just by playing around with these tokens and turning on word probability):
What we’re now finding is that there’s a “continuum of glitchiness”. Some tokens glitch worse/harder than others in a way that I’ve devised an ad-hoc metric for (research report coming soon). There are a lot of “mildly glitchy” tokens that GPT-3 will try to avoid repeating which look like “velength” and “oldemort” (obviously parts of longer, familiar words, rarely seen isolated in text). There’s a long list of these in Part II of this post. I’d not seen “ocobo” or “oldemort” yet, but I’m systematically running tests on the whole vocabulary.
I think I found some weird ones that I haven’t found anyone else document yet (just by playing around with these tokens and turning on word probability):
‘ocobo’, ‘velength’, ‘iannopoulos’, ′ oldemort’, ‘<|endoftext|>‘, ’ ii’
What we’re now finding is that there’s a “continuum of glitchiness”. Some tokens glitch worse/harder than others in a way that I’ve devised an ad-hoc metric for (research report coming soon). There are a lot of “mildly glitchy” tokens that GPT-3 will try to avoid repeating which look like “velength” and “oldemort” (obviously parts of longer, familiar words, rarely seen isolated in text). There’s a long list of these in Part II of this post. I’d not seen “ocobo” or “oldemort” yet, but I’m systematically running tests on the whole vocabulary.