This suggests that without much more data we don’t get much better token prediction, but arguably modern quality of token prediction is already more than sufficient for AGI, we’ve reached token prediction overhang. It’s some other things that are missing, which won’t be resolved with better token prediction. (And it seems there are still ways of improving token prediction a fair bit, but again possibly irrelevant for timelines.)
This suggests that without much more data we don’t get much better token prediction, but arguably modern quality of token prediction is already more than sufficient for AGI, we’ve reached token prediction overhang. It’s some other things that are missing, which won’t be resolved with better token prediction. (And it seems there are still ways of improving token prediction a fair bit, but again possibly irrelevant for timelines.)