I now think it doesn’t work easily, because the training on written language doesn’t have enough examples of people explicitly stating their cognitive steps in applying System 2 reasoning.
The cognitive steps are still part of the hidden structure that generated the data. That GPT-4 level models are unable to capture them is not necessarily evidence that it’s very hard. They’ve just breached the reading comprehension threshold, started to reliably understand most nuanced meaning directly given in the text.
Only in second half of 2024 there’s now enough compute to start experimenting with scale significantly beyond GPT-4 level (with possible recent results still hidden within frontier labs). Before that there wasn’t opportunity to see if something else starts appearing just after GPT-4 scale, so absence of such evidence isn’t yet evidence of absence, that additional currently-absent capabilities aren’t within easy reach. It’s been 2 years at about the same scale of base models, but that isn’t evidence that additional scale stops helping in crucial ways, as no experiments with significant additional scale have been performed in those 2 years.
I totally agree. Natural language datasets do have the right information embedded in them; it’s just obscured by a lot of other stuff. Compute alone might be enough to bring it out.
Part of my original hypothesis was that even a small improvement in the base model might be enough to make scaffolded System 2 type thinking very effective. It’s hard to guess when a system could get past the threshold of having more thinking work better, like it does for humans (with diminishing returns). It could come frome a small improvement in the scaffolding, or a small improvement in memory systems, or even from better feedback from outside sources (e.g., using web searches and better distinguishing good from bad information).
All of those factors are critical in human thinking, and our abilities are clearly a nonlinear product of separate cognitive capacities. That’s why I expect improvements in any or all of those dimensions to eventually lead to human-plus fluid intelligence. And since efforts are underway on each of those dimensions, I’d guess we see that level sooner than later. Two years is my median guess for human level reasoning on most problems, maybe all. But we might still not have good online learning allowing, for a relevant instance, for the system to be trained on any arbitrary job and to then do it competently. Fortunately I expect it to scale past human level at a relatively slow pace from there, giving us a few more years to get our shit together once we’re staring roughly human-equivalent agents in the face and so start to take the potentials seriously.
The cognitive steps are still part of the hidden structure that generated the data. That GPT-4 level models are unable to capture them is not necessarily evidence that it’s very hard. They’ve just breached the reading comprehension threshold, started to reliably understand most nuanced meaning directly given in the text.
Only in second half of 2024 there’s now enough compute to start experimenting with scale significantly beyond GPT-4 level (with possible recent results still hidden within frontier labs). Before that there wasn’t opportunity to see if something else starts appearing just after GPT-4 scale, so absence of such evidence isn’t yet evidence of absence, that additional currently-absent capabilities aren’t within easy reach. It’s been 2 years at about the same scale of base models, but that isn’t evidence that additional scale stops helping in crucial ways, as no experiments with significant additional scale have been performed in those 2 years.
I totally agree. Natural language datasets do have the right information embedded in them; it’s just obscured by a lot of other stuff. Compute alone might be enough to bring it out.
Part of my original hypothesis was that even a small improvement in the base model might be enough to make scaffolded System 2 type thinking very effective. It’s hard to guess when a system could get past the threshold of having more thinking work better, like it does for humans (with diminishing returns). It could come frome a small improvement in the scaffolding, or a small improvement in memory systems, or even from better feedback from outside sources (e.g., using web searches and better distinguishing good from bad information).
All of those factors are critical in human thinking, and our abilities are clearly a nonlinear product of separate cognitive capacities. That’s why I expect improvements in any or all of those dimensions to eventually lead to human-plus fluid intelligence. And since efforts are underway on each of those dimensions, I’d guess we see that level sooner than later. Two years is my median guess for human level reasoning on most problems, maybe all. But we might still not have good online learning allowing, for a relevant instance, for the system to be trained on any arbitrary job and to then do it competently. Fortunately I expect it to scale past human level at a relatively slow pace from there, giving us a few more years to get our shit together once we’re staring roughly human-equivalent agents in the face and so start to take the potentials seriously.