Interesting! I’ve not seen it make reference to ‘<’ and ‘>’ before.
I just searched all 50257 tokens, and the only ones containing both ‘<’ and ‘>’ are
6927 ><
12240 ></
22039 “><
23984 “></
28725 ><
50256 <|endoftext|>
So it seems that 50256 may be relevant. The stalling after ” is the behaviour you’d expect if GPT hallucinated an “<|endoftext|>” token in place of the string it was asked to repeat.
Please keep experimenting and let us know what you find!
Interesting! I’ve not seen it make reference to ‘<’ and ‘>’ before.
I just searched all 50257 tokens, and the only ones containing both ‘<’ and ‘>’ are
6927 ><
12240 ></
22039 “><
23984 “></
28725 ><
50256 <|endoftext|>
So it seems that 50256 may be relevant. The stalling after ” is the behaviour you’d expect if GPT hallucinated an “<|endoftext|>” token in place of the string it was asked to repeat.
Please keep experimenting and let us know what you find!