Assuming that the “start with lots of models then remove the bad ones” picture is accurate, the fact that NNs work in practice implies that the initial set of models matters a lot. The vast majority of models consistent with the data would not generalize well to future data points, unless the models are biased towards models-which-generalize-well (simple models, for instance) from the start.
Assuming that the “start with lots of models then remove the bad ones” picture is accurate, the fact that NNs work in practice implies that the initial set of models matters a lot. The vast majority of models consistent with the data would not generalize well to future data points, unless the models are biased towards models-which-generalize-well (simple models, for instance) from the start.