I enjoyed this post, which feels to me part of a cluster of recent posts pointing out that the current LLM architecture is showing some limitations, that future AI capabilities will likely be quite jagged (thus more complementary to human labor, rather than perfectly substituting for labor as a “drop-in remote worker”), and that a variety of skills around memory, long-term planning, agenticness, etc, seem like like important bottlenecks.
Much of this stuff seems right to me. The jaggedness of AI capabilities, in particular, seems like something that we should’ve spotted much sooner (indeed, it feels like we could’ve gotten most of the way just based on first-principles reasoning), but which has been obscured by the use of helpful abstractions like “AGI” / “human level AI”, or even more rigorous formulations like “when X% of tasks in the economy have been automated”.
I also agree that it’s hard to envision AI transforming the world without a more coherent sense of agency / ability to play pokemon / etc, although I’m agnostic over whether we’ll be able to imbue LLMs with agency via tinkering with scaffolds and training with RL (as discussed elsewhere in this comment thread). At least mild versions of agency seem pretty doable with RL—just train on a bunch of videogames and web-browsing tasks, and I expect AI to get pretty good at completing videogames and web tasks. But whether that’ll scale all the way to being able to manage large software projects and do people’s entire jobs autonomously, I dunno.
If math is solved, though, I don’t know how to estimate the consequences, and it might invalidate the rest of my predictions.
...there’s an explicit carve-out for ??? consequences if math is solved
This got me curious, so I talked to Claude about it. Unfortunately it seems like some of the biggest real-world impacts of “solving math” might come in the form of very significant AI algorithmic improvements, which might obviate some of your other points! (Also: the state of cybersecurity might be thrown into chaos, quant trading would get much more powerful albeit not infinitely powerful, assorted scientific tools could see big improvments.) Here is my full conversation; for the most interesting bit, scroll down to Claude’s final response (ctrl-f for “Category 1: Direct Mathematical Optimization).
I enjoyed this post, which feels to me part of a cluster of recent posts pointing out that the current LLM architecture is showing some limitations, that future AI capabilities will likely be quite jagged (thus more complementary to human labor, rather than perfectly substituting for labor as a “drop-in remote worker”), and that a variety of skills around memory, long-term planning, agenticness, etc, seem like like important bottlenecks.
(Some other posts in this category include this one about Claude’s abysmal Pokemon skills, and the section called “What I suspect AI labs will struggle with in the near term” in this post from Epoch).
Much of this stuff seems right to me. The jaggedness of AI capabilities, in particular, seems like something that we should’ve spotted much sooner (indeed, it feels like we could’ve gotten most of the way just based on first-principles reasoning), but which has been obscured by the use of helpful abstractions like “AGI” / “human level AI”, or even more rigorous formulations like “when X% of tasks in the economy have been automated”.
I also agree that it’s hard to envision AI transforming the world without a more coherent sense of agency / ability to play pokemon / etc, although I’m agnostic over whether we’ll be able to imbue LLMs with agency via tinkering with scaffolds and training with RL (as discussed elsewhere in this comment thread). At least mild versions of agency seem pretty doable with RL—just train on a bunch of videogames and web-browsing tasks, and I expect AI to get pretty good at completing videogames and web tasks. But whether that’ll scale all the way to being able to manage large software projects and do people’s entire jobs autonomously, I dunno.
This got me curious, so I talked to Claude about it. Unfortunately it seems like some of the biggest real-world impacts of “solving math” might come in the form of very significant AI algorithmic improvements, which might obviate some of your other points! (Also: the state of cybersecurity might be thrown into chaos, quant trading would get much more powerful albeit not infinitely powerful, assorted scientific tools could see big improvments.) Here is my full conversation; for the most interesting bit, scroll down to Claude’s final response (ctrl-f for “Category 1: Direct Mathematical Optimization).