I think we’re mostly agreeing, I’ve gotten less and less convinced that “LLMs are doomed” in the last few days, that’s a much stronger statement than I actually believe right now. What I mean by “generalisation” is basically what you mean by learning the pattern of human thought from the text, in my mind producing good outputs on inputs with low probability is by definition “generalisation”.
I agree that none of these arguments strictly prevent this “learning the structure of human thoughts” from happening, but I would still be somewhat surprised if it did, since neural networks in other contexts like vision and robotics don’t seem to generalise this far, but maybe text really is special, as the past few months seem to indicate.
I think we’re mostly agreeing, I’ve gotten less and less convinced that “LLMs are doomed” in the last few days, that’s a much stronger statement than I actually believe right now. What I mean by “generalisation” is basically what you mean by learning the pattern of human thought from the text, in my mind producing good outputs on inputs with low probability is by definition “generalisation”.
I agree that none of these arguments strictly prevent this “learning the structure of human thoughts” from happening, but I would still be somewhat surprised if it did, since neural networks in other contexts like vision and robotics don’t seem to generalise this far, but maybe text really is special, as the past few months seem to indicate.