While I agree that existing learning theory leaves much to be desired, I don’t think your previous posts have presented strong enough evidence for the superiority of singular learning theory that we should “scrap the whole field and start over”. The gold standard for any theory of deep learning is “actually being able to predict non-trivial properties of real networks” and AFAIK SLT does not yet meet this standard[1]. Which is not to say that it shouldn’t be pursued—let a thousand flowers bloom. But I think your post has a vibe of “the SLT is the only worthwhile avenue to explore” which I don’t think is well-supported by the evidence. For e.g. here are some papers which are part of research programs which seem at least as promising as SLT as potential avenues to explore to me.
While I agree that existing learning theory leaves much to be desired, I don’t think your previous posts have presented strong enough evidence for the superiority of singular learning theory that we should “scrap the whole field and start over”. The gold standard for any theory of deep learning is “actually being able to predict non-trivial properties of real networks” and AFAIK SLT does not yet meet this standard[1]. Which is not to say that it shouldn’t be pursued—let a thousand flowers bloom. But I think your post has a vibe of “the SLT is the only worthwhile avenue to explore” which I don’t think is well-supported by the evidence. For e.g. here are some papers which are part of research programs which seem at least as promising as SLT as potential avenues to explore to me.
less so than the NTK which you deride, for instance!
I agree with you that I haven’t presented enough evidence! Which is why this is the first part in a six-part sequence.