I’m very much in agreement with you overall. I personally expect that the strongest general models of 2026 will be clearly capable of recursive improvement via both ML research and scaffolding development.
I do think that this particular point you make needs adjusting:
We can also check predictions made about AI from three years ago to see if we’ve overshot or undershot expectations. In 2021, Jacob Steinhardt’s group (who also developed some of the benchmarks used) commissioned a forecast which included SOTA estimates on AI in 2024.
I’m not doing great at understanding the viewpoint of the AI-fizzle crowd, but I think it would be more fair to them to analyze these forecasts in terms of expected-improvement-per-dollar versus actual-improvement-per-dollar. My guess is that 2021 forecasts anticipated less money invested, and that one could argue that per-dollar gains have been lower than expected.
I don’t think this fundamentally changes anything overall, I’m just trying to steelman.
You might be right—and whether the per-dollar gains were higher or lower than expected would be interesting to know—but I just don’t have any good information on this! If I’d thought of the possibility, I would have added it in Footnote 23 as another speculation, but I don’t think what I said is misleading or wrong.
For what it’s worth, in a one year review from Jacob Steinhardt, increased investment isn’t mentioned as an explanation for why the forecasts undershot.
I’m very much in agreement with you overall. I personally expect that the strongest general models of 2026 will be clearly capable of recursive improvement via both ML research and scaffolding development.
I do think that this particular point you make needs adjusting:
I’m not doing great at understanding the viewpoint of the AI-fizzle crowd, but I think it would be more fair to them to analyze these forecasts in terms of expected-improvement-per-dollar versus actual-improvement-per-dollar. My guess is that 2021 forecasts anticipated less money invested, and that one could argue that per-dollar gains have been lower than expected.
I don’t think this fundamentally changes anything overall, I’m just trying to steelman.
You might be right—and whether the per-dollar gains were higher or lower than expected would be interesting to know—but I just don’t have any good information on this! If I’d thought of the possibility, I would have added it in Footnote 23 as another speculation, but I don’t think what I said is misleading or wrong.
For what it’s worth, in a one year review from Jacob Steinhardt, increased investment isn’t mentioned as an explanation for why the forecasts undershot.