Thanks, yes, I should have mentioned ‘unhobbling’ in that sentence, have added.
I’ve gone back and added my thoughts on unhobbling in a footnote: “Chollet Aschenbrenner also discusses ‘unhobbling’, which he describes as ‘fixing obvious ways in which models are hobbled by default, unlocking latent capabilities and giving them tools, leading to step-changes in usefulness’. He breaks that down into categories here. Scaffolding and tooling I discuss here; RHLF seems unlikely to help with fundamental reasoning issues. Increased context length serves roughly as a kind of scaffolding for purposes of this discussion. ‘Posttraining improvements’ is too vague to really evaluate. But note that his core claim (the graph here) ‘shows only the scaleup in base models; “unhobblings” are not pictured’.”
But I don’t see from where do you see such high probability (>5%) of scaffolding not working
Frankly I’d be hesitant to put > 95% on almost any claims on this topic. My strongest reason for suspecting that scaffolding might not work to get LLMs to AGI is updating on the fact that it doesn’t seem to have become a useful approach yet despite many people’s efforts (and despite the lack of obvious blockers). I certainly expect scaffolding to improve over where it is now, but I haven’t seen much reason to believe that it’ll enable planning and general reasoning capabilities that are enormously greater than LLMs’ base capabilities.
… I mean whatever will work can be retroactively called “scaffolding”, even if it will be in the “one more major breakthrough” category
What I mean by scaffolding here is specifically wrapping the model in a broader system consisting of some combination of goal-direction, memory, and additional tools that the system can use (not ones that the model calls; I’d put those in the ‘tooling’ category), with a central outer loop that makes calls to the model. Breakthroughs resulting in better modelswouldn’t count on my definition.
Thanks for the clarification, I don’t share the intuition this will prove harder than other hard software engineering challenges in non-AI areas that weren’t solved in months but were solved in years and not decades, but other than “broad baseline is more significant than narrow evidence for me” I don’t have anything more concrete to share.
A note until fixed: Chollet also discusses ‘unhobbling’ → Aschenbrenner also discusses ‘unhobbling’
I think the shift of my intuition over the past year has looked something like: a) (a year ago) LLMs seem really smart and general (especially given all the stuff they unexpectedly learned like translation), but they lack goals and long-term memory, I bet if we give them that they’ll be really impressive. b) Oh, huh, if we add goals and long-term memory they don’t actually do that well. c) Oh, huh, they fail at stuff that seems pretty basic relative to how smart and general they seem. d) OK, probably we should question our initial impression of how smart and general they are. I realize that’s not really a coherent argument; just trying to give a sense of the overall shape of why I’ve personally gotten more skeptical.
I’ve gone back and added my thoughts on unhobbling in a footnote: “
CholletAschenbrenner also discusses ‘unhobbling’, which he describes as ‘fixing obvious ways in which models are hobbled by default, unlocking latent capabilities and giving them tools, leading to step-changes in usefulness’. He breaks that down into categories here. Scaffolding and tooling I discuss here; RHLF seems unlikely to help with fundamental reasoning issues. Increased context length serves roughly as a kind of scaffolding for purposes of this discussion. ‘Posttraining improvements’ is too vague to really evaluate. But note that his core claim (the graph here) ‘shows only the scaleup in base models; “unhobblings” are not pictured’.”Frankly I’d be hesitant to put > 95% on almost any claims on this topic. My strongest reason for suspecting that scaffolding might not work to get LLMs to AGI is updating on the fact that it doesn’t seem to have become a useful approach yet despite many people’s efforts (and despite the lack of obvious blockers). I certainly expect scaffolding to improve over where it is now, but I haven’t seen much reason to believe that it’ll enable planning and general reasoning capabilities that are enormously greater than LLMs’ base capabilities.
What I mean by scaffolding here is specifically wrapping the model in a broader system consisting of some combination of goal-direction, memory, and additional tools that the system can use (not ones that the model calls; I’d put those in the ‘tooling’ category), with a central outer loop that makes calls to the model. Breakthroughs resulting in better models wouldn’t count on my definition.
Thanks for the clarification, I don’t share the intuition this will prove harder than other hard software engineering challenges in non-AI areas that weren’t solved in months but were solved in years and not decades, but other than “broad baseline is more significant than narrow evidence for me” I don’t have anything more concrete to share.
A note until fixed:Chollet also discusses ‘unhobbling’→Aschenbrenner also discusses ‘unhobbling’I think the shift of my intuition over the past year has looked something like: a) (a year ago) LLMs seem really smart and general (especially given all the stuff they unexpectedly learned like translation), but they lack goals and long-term memory, I bet if we give them that they’ll be really impressive. b) Oh, huh, if we add goals and long-term memory they don’t actually do that well. c) Oh, huh, they fail at stuff that seems pretty basic relative to how smart and general they seem. d) OK, probably we should question our initial impression of how smart and general they are. I realize that’s not really a coherent argument; just trying to give a sense of the overall shape of why I’ve personally gotten more skeptical.