As an aside, I think this is more about data instead of “how easy is it to implement.”
This seems confused to me—I’m not sure that there’s a meaningful sense in which you can say one of data vs. inductive biases matters “more.” They are both absolutely essential, and you can’t talk about what algorithm will be learned by a machine learning system unless you are engaging both with the nature of the data and the nature of the inductive biases, since if you only fix one and not the other you can learn essentially any algorithm.
Furthermore, a vision system modeled after primate vision also generalized based on texture, which is further evidence against ANN-specific architectural biases (like conv layers) explaining the discrepancy.
To be clear, I’m not saying that the inductive biases that matter here are necessarily unique to ANNs. In fact, they can’t be: by Occam’s razor, simplicity bias is what gets you good generalization, and since both human neural networks and artificial neural networks can often achieve good generalization, they have to be both be using a bunch of shared simplicity bias.
The problem is that pure simplicity bias doesn’t actually get you alignment. So even if humans and AIs share 99% of inductive biases, what they’re sharing is just the obvious simplicity bias stuff that any system capable of generalizing from real-world data has to share.
This seems confused to me—I’m not sure that there’s a meaningful sense in which you can say one of data vs. inductive biases matters “more.” They are both absolutely essential, and you can’t talk about what algorithm will be learned by a machine learning system unless you are engaging both with the nature of the data and the nature of the inductive biases, since if you only fix one and not the other you can learn essentially any algorithm.
To be clear, I’m not saying that the inductive biases that matter here are necessarily unique to ANNs. In fact, they can’t be: by Occam’s razor, simplicity bias is what gets you good generalization, and since both human neural networks and artificial neural networks can often achieve good generalization, they have to be both be using a bunch of shared simplicity bias.
The problem is that pure simplicity bias doesn’t actually get you alignment. So even if humans and AIs share 99% of inductive biases, what they’re sharing is just the obvious simplicity bias stuff that any system capable of generalizing from real-world data has to share.