For what it’s worth, I’m at least somewhat an LLM-plateau-ist—on balance at least somewhat dubious we get AGI from models in which 99% of compute is spent on next-word prediction in big LLMs. I really think Nostalgebrists take has merit and the last few months have made me think it has more merit. Yann LeCunn’s “LLMs are an off-ramp to AGI” might come back to show his forsight. Etc etc.
But it isn’t just LLM progress which has hinged on big quantities of compute. Everything in deep learning—ResNets, vision Transformers, speech-to-text, text-to-speech, AlphaGo, EfficientZero, Dota5, VPT, and so on—has used more and more compute. I think at least some of this deep learning stuff is an important step to human-like intelligence, which is why I think this is good evidence against Yudkowsky
If you think none of the DL stuff is a step, then you can indeed maintain the compute doesn’t matter, of course, and that I am horribly wrong. But if you think the DL stuff is an important step, it becomes more difficult to maintain.
For what it’s worth, I’m at least somewhat an LLM-plateau-ist—on balance at least somewhat dubious we get AGI from models in which 99% of compute is spent on next-word prediction in big LLMs. I really think Nostalgebrists take has merit and the last few months have made me think it has more merit. Yann LeCunn’s “LLMs are an off-ramp to AGI” might come back to show his forsight. Etc etc.
But it isn’t just LLM progress which has hinged on big quantities of compute. Everything in deep learning—ResNets, vision Transformers, speech-to-text, text-to-speech, AlphaGo, EfficientZero, Dota5, VPT, and so on—has used more and more compute. I think at least some of this deep learning stuff is an important step to human-like intelligence, which is why I think this is good evidence against Yudkowsky
If you think none of the DL stuff is a step, then you can indeed maintain the compute doesn’t matter, of course, and that I am horribly wrong. But if you think the DL stuff is an important step, it becomes more difficult to maintain.