For an even more extreme example, Linear Regression is a large “vector of floating points” that’s easy enough to prove things about proofs are assigned as homework questions for introductory Linear Algebra/Statistics classes.
I also think that we’ve had more significantly more theoretical progress on ANNs than I would’ve predicted ~5 years ago. For example, the theory of wide two layer neural networks has basically been worked out, as had the theory of infinitely deep neural networks, and the field has made significant progress understanding why we might expect the critical points/local minima gradient descent finds in ReLU networks to generalize. (Though none of this work is currently good enough to inform practice, I think?)
+1 to both points.
For an even more extreme example, Linear Regression is a large “vector of floating points” that’s easy enough to prove things about proofs are assigned as homework questions for introductory Linear Algebra/Statistics classes.
I also think that we’ve had more significantly more theoretical progress on ANNs than I would’ve predicted ~5 years ago. For example, the theory of wide two layer neural networks has basically been worked out, as had the theory of infinitely deep neural networks, and the field has made significant progress understanding why we might expect the critical points/local minima gradient descent finds in ReLU networks to generalize. (Though none of this work is currently good enough to inform practice, I think?)