Making it stronger means more weights so the regularization should push against it, UNLESS you can simultaneously delete or dampen weights from the memorized answer part, right?
I think this does happen (and is very surprising to me!). If you look at the excluded loss section, I ablate the model’s ability to use one component of the generalising algorithm, in a way that shouldn’t affect the memorising algorithm (much), and see the damage of this ablation rise smoothly over training. I hypothesise that it’s dampening memorisation weights simulatenously, though haven’t dug deep enough to be confident. Regardless, it clearly seems to be doing some kind of interpolation—I have a lot of progress measures that all (both qualitatively and quantitatively) show clear progress towards generalisation pre grokking.
I think this does happen (and is very surprising to me!). If you look at the excluded loss section, I ablate the model’s ability to use one component of the generalising algorithm, in a way that shouldn’t affect the memorising algorithm (much), and see the damage of this ablation rise smoothly over training. I hypothesise that it’s dampening memorisation weights simulatenously, though haven’t dug deep enough to be confident. Regardless, it clearly seems to be doing some kind of interpolation—I have a lot of progress measures that all (both qualitatively and quantitatively) show clear progress towards generalisation pre grokking.