Sure, but it’s unclear whether these inductive biases are necessary.
My understanding is that they’re necessary even in principle, since there are an unbounded number of functions that fit any finite set of points. Even AIXI has a strong inductive bias, toward programs with the lowest Kolmogorov complexity.
Yeah, I agree that some inductive bias is probably necessary. But not all inductive biases are equal; some are much more general than others, and in particular I claim that ‘narrow’ inductive biases (e.g. specializing architectures to match domains) probably have net ~zero benefit compared to those learned from data
My understanding is that they’re necessary even in principle, since there are an unbounded number of functions that fit any finite set of points. Even AIXI has a strong inductive bias, toward programs with the lowest Kolmogorov complexity.
Yeah, I agree that some inductive bias is probably necessary. But not all inductive biases are equal; some are much more general than others, and in particular I claim that ‘narrow’ inductive biases (e.g. specializing architectures to match domains) probably have net ~zero benefit compared to those learned from data