To be honest, this argument makes me even more confident in short times. I feel like the focus on scaling and data requirements completely miss the point. GPT-4 is already much smarter than I am in the ways that it is smart. Adding more scale and data might continue to make it better, but it doesn’t need to be better in that way to become transformative. The problem is the limitations—limited context window, no continual learning, text encoding issues, no feedback loop REPL wrapper creating agency, expensive to run, robotics is lagging. These are not problems that will take decades to solve, they will take years, if not months.
Gary Marcus’s new goalpost is that the AI has to invent new science with only training data from before a specific year. I can’t do that! I couldn’t do that no matter how much training data I had. Am I a general intelligence Gary? I feel like this is all some weird cope.
To be clear, I’m not blind to the fact that LLMs are following the same hype cycle that other technologies have gone through. I’m sure there will be some media narrative in a year or so like “AI was going to take all our jobs, but that hasn’t happened yet, it was just hype.” Meanwhile, researchers (which now includes essentially everyone who knows how to install python) will fix the limitations and make these systems ever more powerful.
I am highly confident that current AI technologies, without any more scale or data[1], will be able to do any economically relevant task, within the next 10 years.
I disagree with the model a little, but agree that not much scaling is required further, and the things that will make an AGI viable, like a memory that can model arbitrarily long/complicated problems is essentially only blocked by boring engineering, which means we will eventually be able to create AGI pretty soon.
To be honest, this argument makes me even more confident in short times. I feel like the focus on scaling and data requirements completely miss the point. GPT-4 is already much smarter than I am in the ways that it is smart. Adding more scale and data might continue to make it better, but it doesn’t need to be better in that way to become transformative. The problem is the limitations—limited context window, no continual learning, text encoding issues, no feedback loop REPL wrapper creating agency, expensive to run, robotics is lagging. These are not problems that will take decades to solve, they will take years, if not months.
Gary Marcus’s new goalpost is that the AI has to invent new science with only training data from before a specific year. I can’t do that! I couldn’t do that no matter how much training data I had. Am I a general intelligence Gary? I feel like this is all some weird cope.
To be clear, I’m not blind to the fact that LLMs are following the same hype cycle that other technologies have gone through. I’m sure there will be some media narrative in a year or so like “AI was going to take all our jobs, but that hasn’t happened yet, it was just hype.” Meanwhile, researchers (which now includes essentially everyone who knows how to install python) will fix the limitations and make these systems ever more powerful.
I am highly confident that current AI technologies, without any more scale or data[1], will be able to do any economically relevant task, within the next 10 years.
We will need new training data, specifically for robotics, but we won’t need more data. These systems are already smart enough.
I disagree with the model a little, but agree that not much scaling is required further, and the things that will make an AGI viable, like a memory that can model arbitrarily long/complicated problems is essentially only blocked by boring engineering, which means we will eventually be able to create AGI pretty soon.