random fun experiment: accuracy of GPT-4 on “Q: What is 1 + 1 + 1 + 1 + …?\nA:”
blue: highest logprob numerical token
orange: y = x
...I am suddenly really curious what the accuracy of humans on that is.
‘Can you do Addition?’ the White Queen asked. ‘What’s one and one and one and one and one and one and one and one and one and one?’
‘I don’t know,’ said Alice. ‘I lost count.’
This is a cool idea. I wonder how it’s able to do 100, 150, and 200 so well. I also wonder what are the exact locations of the other spikes?
Oh, I see your other graph now. So it just always guesses 100 for everything in the vicinity of 100.
random fun experiment: accuracy of GPT-4 on “Q: What is 1 + 1 + 1 + 1 + …?\nA:”
blue: highest logprob numerical token
orange: y = x
...I am suddenly really curious what the accuracy of humans on that is.
‘Can you do Addition?’ the White Queen asked. ‘What’s one and one and one and one and one and one and one and one and one and one?’
‘I don’t know,’ said Alice. ‘I lost count.’
This is a cool idea. I wonder how it’s able to do 100, 150, and 200 so well. I also wonder what are the exact locations of the other spikes?
Oh, I see your other graph now. So it just always guesses 100 for everything in the vicinity of 100.