Thanks, that helps. Perhaps an example would be: A purely feed-forward neural network might be “blind” to algorithms that are kolmogorov-simple but which involve repeatedly performing the same procedure a bunch of times (even if it is technically big enough to contain such an algorithm). So the simplicity bias of said network would be importantly different from kolmogorov complexity.
Thanks, that helps. Perhaps an example would be: A purely feed-forward neural network might be “blind” to algorithms that are kolmogorov-simple but which involve repeatedly performing the same procedure a bunch of times (even if it is technically big enough to contain such an algorithm). So the simplicity bias of said network would be importantly different from kolmogorov complexity.
That’s exactly right. That exact example would be a case of high circuit size but low Kolmogorov complexity.