Regardless, its amazing how simple DNNs are. People have been working on computer vision and AI for about 60 years, and then a program like this comes along which is only around 500 lines of code, conceptually simple enough to explain to anyone with a reasonable mathematical background, but can nevertheless beat humans at a reasonable range of tasks.
I get the impression it’s a hardware issue. See for example http://nautil.us/issue/21/information/the-man-who-tried-to-redeem-the-world-with-logic—McCulloch & Pitts invented neural networks almost before digital computers existed* and he was working on “three-dimensional neural networks”. They didn’t invent backpropagation, I don’t think, but even if they had, how would they have run, much less trained, the state of the art many-layer neural networks with millions of nodes and billions of connections like we’re seeing these days? What those 60 years of work gets you is a lot of specialized algorithms which don’t reach human-parity but at least are computable on the hardware of that day.
* depends on what exactly you consider the first digital computer and how long before the key publication you date their breakthrough.
Can confirm that hardware (and data!) are the two main culprits here. The actual learning algorithms haven’t changed much since the mid 1980s, but computers have gotten many times faster, GPUs are 30-100x faster still, and the amount of data has similarly increased by several orders of magnitude.
I get the impression it’s a hardware issue. See for example http://nautil.us/issue/21/information/the-man-who-tried-to-redeem-the-world-with-logic—McCulloch & Pitts invented neural networks almost before digital computers existed* and he was working on “three-dimensional neural networks”. They didn’t invent backpropagation, I don’t think, but even if they had, how would they have run, much less trained, the state of the art many-layer neural networks with millions of nodes and billions of connections like we’re seeing these days? What those 60 years of work gets you is a lot of specialized algorithms which don’t reach human-parity but at least are computable on the hardware of that day.
* depends on what exactly you consider the first digital computer and how long before the key publication you date their breakthrough.
Can confirm that hardware (and data!) are the two main culprits here. The actual learning algorithms haven’t changed much since the mid 1980s, but computers have gotten many times faster, GPUs are 30-100x faster still, and the amount of data has similarly increased by several orders of magnitude.