Upvoted and up-concreted your take, I really appreciate experiments like this. That said:
This isn’t necessarily overwhelming evidence of anything, but it might genuinely make my timelines longer. Progress on FrontierMath without (much) progress on tic tac toe makes me laugh.
I’m confused why you think o1 losing the same way in tic tac toe repeatedly shortens your timelines, given that it’s o3 that pushed the FrontierMath SOTA score from 2% to 25% (and o1 was ~1%). I’d agree if it was o3 that did the repeated same-way losing, since that would make your second sentence make sense to me.
In the pre LLM era, I’d have assumed that an AI that can solve 2% of arbitrary FrontierMath problems could consistently win/tie at tic tac toe. Knowing this isn’t the case is interesting. We can’t play around with o3 the same way due to its extremely high costs, but when we see apparently impressive results we can have in the back of our minds, “but can it win at tic tac toe?”
Upvoted and up-concreted your take, I really appreciate experiments like this. That said:
I’m confused why you think o1 losing the same way in tic tac toe repeatedly shortens your timelines, given that it’s o3 that pushed the FrontierMath SOTA score from 2% to 25% (and o1 was ~1%). I’d agree if it was o3 that did the repeated same-way losing, since that would make your second sentence make sense to me.
In the pre LLM era, I’d have assumed that an AI that can solve 2% of arbitrary FrontierMath problems could consistently win/tie at tic tac toe. Knowing this isn’t the case is interesting. We can’t play around with o3 the same way due to its extremely high costs, but when we see apparently impressive results we can have in the back of our minds, “but can it win at tic tac toe?”
That makes more sense, thanks :)