If you could implement the research capability in a way that doesn’t also provide the functionality for scheming, the same way I could implement the “output syntactically correct sentences” capability without providing the general-intelligence functionality, that would work as a disproof of my views.
Yes, but why do you expect this to be hard? As in “much harder than gathering enough hardware”. The shape of the argument seems to me to be “the algorithm humans use for math research is general intelligence is ability to scheme, LLMs are not general, therefore LLMs can’t do it”. But before LLMs we also hadn’t known about the algorithm to do what GPT4 does, the way we know how to generate syntactically correct sentences. If you can’t think of an algorithm, why automatically expect GPT-6 to fail? Even under your model of how LLMs work (which may be biased to predict your expected conclusion) its possible that you only need some relatively small number of heuristics to greatly advance math research.
To be clear, my point is not that what you are saying is implausible or counterintuitive. I’m just saying, that, given the stakes, it would be nice if the whole field transitioned to the level of more detailed rigorous justifications with numbers.
What GPT4 does.
Yes, but why do you expect this to be hard? As in “much harder than gathering enough hardware”. The shape of the argument seems to me to be “the algorithm humans use for math research is general intelligence is ability to scheme, LLMs are not general, therefore LLMs can’t do it”. But before LLMs we also hadn’t known about the algorithm to do what GPT4 does, the way we know how to generate syntactically correct sentences. If you can’t think of an algorithm, why automatically expect GPT-6 to fail? Even under your model of how LLMs work (which may be biased to predict your expected conclusion) its possible that you only need some relatively small number of heuristics to greatly advance math research.
To be clear, my point is not that what you are saying is implausible or counterintuitive. I’m just saying, that, given the stakes, it would be nice if the whole field transitioned to the level of more detailed rigorous justifications with numbers.
Well, be the change you wish to see!
I too think it would be incredibly nice, and am working on it. But formalizing cognition is, you know. A major scientific challenge.