I’m pretty concerned about assuming continuity of capability in weights, at least in the strong form that I think you’re relying on.
What I mean is: it might be true that capabilities are continuous in the formal mathematical sense, but if the slope suddenly becomes enormous that’s not much comfort. And there are reasons to expect large slopes for models with memory (because a lot of learning gets done outside of SGD).
I certainly wouldn’t bet the light cone on that assumption! I do think it would very surprising if a single gradient step led to a large increase in capabilities, even with models that do a lot of learning between gradient steps. Would love to see empirical evidence on this.
I’m pretty concerned about assuming continuity of capability in weights, at least in the strong form that I think you’re relying on.
What I mean is: it might be true that capabilities are continuous in the formal mathematical sense, but if the slope suddenly becomes enormous that’s not much comfort. And there are reasons to expect large slopes for models with memory (because a lot of learning gets done outside of SGD).
I certainly wouldn’t bet the light cone on that assumption! I do think it would very surprising if a single gradient step led to a large increase in capabilities, even with models that do a lot of learning between gradient steps. Would love to see empirical evidence on this.