In principle the exercise of sketching a theory and seeing if something like that can be convinced to make more sense is useful, and flaws don’t easily invalidate the exercise even when it’s unclear how to fix them. But I don’t see much hope here?
There’s human sample efficiency, and being smart despite learning on stupid data. With a bit of serial speed advantage and a bit of going beyond average human researcher intelligence, it won’t take long to reproduce that. Then the calculation needs new anchors, and in any case properties of pre-trained LLMs are only briefly relevant if the next few years of blind scaling spit out an AGI, and probably not at all relevant otherwise.
In principle the exercise of sketching a theory and seeing if something like that can be convinced to make more sense is useful, and flaws don’t easily invalidate the exercise even when it’s unclear how to fix them. But I don’t see much hope here?
There’s human sample efficiency, and being smart despite learning on stupid data. With a bit of serial speed advantage and a bit of going beyond average human researcher intelligence, it won’t take long to reproduce that. Then the calculation needs new anchors, and in any case properties of pre-trained LLMs are only briefly relevant if the next few years of blind scaling spit out an AGI, and probably not at all relevant otherwise.