I definitely think that LLMs are ‘smarter than expected’ for many people due to tokenization, if only because they look at tokenization errors, which are so vivid and clear, and then ignore things like GPQA which are arcane and hard to read, and conclude LLMs are stupid. “It can’t even count the letters in ‘strawberry’, obviously this is all bunk.”
I definitely think that LLMs are ‘smarter than expected’ for many people due to tokenization, if only because they look at tokenization errors, which are so vivid and clear, and then ignore things like GPQA which are arcane and hard to read, and conclude LLMs are stupid. “It can’t even count the letters in ‘strawberry’, obviously this is all bunk.”