The outside view is not very good for predicting technology. Every technology has an eternity of not existing, until suddenly one day it exists out of the blue.
Now no one is saying that deep learning is going to be AGI in 10 years. In fact the deep learning experts have been extremely skeptical of AGI in all forms, and are certainly not promoting that view. But I think it’s a very reasonable opinion that it will lead to AGI within the next few decades. And I believe sooner rather than later.
The reasons that ‘this time it is different’:
NNs are extraordinarily general. I don’t think you can say this about other AI approaches. I mean search and planning algorithms are pretty general. But they fall back on needing heuristics to shrink the search space. And how do you learn heuristics? It goes back to being a machine learning problem. And they are starting to solve it. E.g. a deep neural net predicted Go moves made by experts 54% of the time.
The progress you see is a great deal due to computing power advances. Early AI researchers were working with barely any computing power, and a lot of their work reflects that. That’s not to say we have AGI and are just waiting for computers to get fast enough. But computing power allows researchers to experiment and actually do research.
Empirically they have made significant progress on a number of different AI domains. E.g. vision, speech recognition, natural language processing, and Go. A lot of previous AI approaches might have sounded cool in theory, or worked on a single domain, but they could never point to actual success on loads of different AI problems.
It’s more brain like. I know someone will say that they really aren’t anything like the brain. And that’s true, but at a high level there are very similar principles. Learning networks of features and their connections, as opposed to symbolic approaches.
And if you look at the models that are inspired by the brain like HTM, they are sort of converging on similar algorithms. E.g. they say the important part of the cortex is that it’s very sparse and has lateral inhibition. And you see leading researchers propose very similar ideas.
Whereas the stuff they do differently is mostly because they want to follow biological constraints. Like only local interactions, little memory, only single bits of information at a time. And these aren’t restrictions that real computers have too much, so we don’t necessarily need to copy biology in those respects and can do things differently, and even better.
Several of the above claims don’t seem that true to me.
Statistical methods are also very general. And neural nets definitely need heuristics (LSTMs are basically a really good heuristic for getting NNs to train well).
I’m not aware of great success in Go? 54% accuracy is very hard to interpret in a vaccuum in terms of how impressed to be.
When statistical methods displaced logical methods it’s because they led to lots of progress on lots of domains. In fact, the delta from logical to statistical was probably much larger than the delta from classical statistical learning to neural nets.
I consider deep learning to be in the family of statistical methods. The problem with previous statistical methods is that they were shallow and couldn’t learn very complicated functions or structure. No one ever claimed that linear regression would lead to AGI.
I’m not aware of great success in Go? 54% accuracy is very hard to interpret in a vaccuum in terms of how impressed to be.
That narrows the search space to maybe 2 moves or so per board. Which makes heuristic searching algorithms much more practical. You can not only generate good moves and predict what a human will do, but you can combine that with brute force and search much deeper than a human as well.
And neural nets definitely need heuristics
I mean that NNs learn heuristics. They do require heuristics in the learning algorithm, but not ones that are specific to the domain. Whereas search algorithms depend on lots of domain dependent, manually created heuristics.
The outside view is not very good for predicting technology. Every technology has an eternity of not existing, until suddenly one day it exists out of the blue.
Now no one is saying that deep learning is going to be AGI in 10 years. In fact the deep learning experts have been extremely skeptical of AGI in all forms, and are certainly not promoting that view. But I think it’s a very reasonable opinion that it will lead to AGI within the next few decades. And I believe sooner rather than later.
The reasons that ‘this time it is different’:
NNs are extraordinarily general. I don’t think you can say this about other AI approaches. I mean search and planning algorithms are pretty general. But they fall back on needing heuristics to shrink the search space. And how do you learn heuristics? It goes back to being a machine learning problem. And they are starting to solve it. E.g. a deep neural net predicted Go moves made by experts 54% of the time.
The progress you see is a great deal due to computing power advances. Early AI researchers were working with barely any computing power, and a lot of their work reflects that. That’s not to say we have AGI and are just waiting for computers to get fast enough. But computing power allows researchers to experiment and actually do research.
Empirically they have made significant progress on a number of different AI domains. E.g. vision, speech recognition, natural language processing, and Go. A lot of previous AI approaches might have sounded cool in theory, or worked on a single domain, but they could never point to actual success on loads of different AI problems.
It’s more brain like. I know someone will say that they really aren’t anything like the brain. And that’s true, but at a high level there are very similar principles. Learning networks of features and their connections, as opposed to symbolic approaches.
And if you look at the models that are inspired by the brain like HTM, they are sort of converging on similar algorithms. E.g. they say the important part of the cortex is that it’s very sparse and has lateral inhibition. And you see leading researchers propose very similar ideas.
Whereas the stuff they do differently is mostly because they want to follow biological constraints. Like only local interactions, little memory, only single bits of information at a time. And these aren’t restrictions that real computers have too much, so we don’t necessarily need to copy biology in those respects and can do things differently, and even better.
Several of the above claims don’t seem that true to me.
Statistical methods are also very general. And neural nets definitely need heuristics (LSTMs are basically a really good heuristic for getting NNs to train well).
I’m not aware of great success in Go? 54% accuracy is very hard to interpret in a vaccuum in terms of how impressed to be.
When statistical methods displaced logical methods it’s because they led to lots of progress on lots of domains. In fact, the delta from logical to statistical was probably much larger than the delta from classical statistical learning to neural nets.
I consider deep learning to be in the family of statistical methods. The problem with previous statistical methods is that they were shallow and couldn’t learn very complicated functions or structure. No one ever claimed that linear regression would lead to AGI.
That narrows the search space to maybe 2 moves or so per board. Which makes heuristic searching algorithms much more practical. You can not only generate good moves and predict what a human will do, but you can combine that with brute force and search much deeper than a human as well.
I mean that NNs learn heuristics. They do require heuristics in the learning algorithm, but not ones that are specific to the domain. Whereas search algorithms depend on lots of domain dependent, manually created heuristics.