since we train exclusively on loss and effectively only use our inductive biases as a tiebreaker
If that were true, I’d buy the story presented in double descent. But we don’t do that; we regularize throughout training! The loss usually includes an explicit term that penalizes the L2 norm of the weights, and that loss is evaluated and trained against throughout training, and across models, and regardless of dataset size.
It might be that the inductive biases are coming from some other method besides regularization (especially since some of the experiments are done without regularization iirc). But even then, to be convinced of this story, I’d want to see some explanation of how in terms of the training dynamics the inductive biases act as a tiebreaker, and why that explanation doesn’t do anything before the interpolation threshold.
Reading your comment again, the first three sentences seem different from the last two sentences. My response above is responding to the last two sentences; I’m not sure if you mean something different by the first three sentences.
If that were true, I’d buy the story presented in double descent. But we don’t do that; we regularize throughout training! The loss usually includes an explicit term that penalizes the L2 norm of the weights, and that loss is evaluated and trained against throughout training, and across models, and regardless of dataset size.
It might be that the inductive biases are coming from some other method besides regularization (especially since some of the experiments are done without regularization iirc). But even then, to be convinced of this story, I’d want to see some explanation of how in terms of the training dynamics the inductive biases act as a tiebreaker, and why that explanation doesn’t do anything before the interpolation threshold.
Reading your comment again, the first three sentences seem different from the last two sentences. My response above is responding to the last two sentences; I’m not sure if you mean something different by the first three sentences.