Deep Learning is the latest thing in AI. I predict that it will be exactly as successful at achieving AGI as all previous latest things. By which I mean that in 10 years it will be just another chapter in the latest edition of Russell and Norvig.
Outside View only. That’s the way it’s always worked out before, and I’m not seeing anything specific to Deep Learning to suggest that this time, it will be different. But I am not a professional in this field.
So, some Inside View reasons to think this time might be different:
The results look better, and in particular, some of Google’s projects are reproducing high-level quirks of the human visual cortex.
The methods can absorb far larger amounts of computing power. Previous approaches could not, which makes sense as we didn’t have the computing power for them to absorb at the time, but the human brain does appear to be almost absurdly computation-heavy. Moore’s Law is producing a difference in kind.
That said, I (and most AI researchers, I believe) would agree that deep recurrent networks are only part of the puzzle. The neat thing is, they do appear to be part of the puzzle, which is more than you could say about e.g. symbolic logic; human minds don’t run on logic at all. We’re making progress, and I wouldn’t be surprised if deep learning is part of the first AGI.
some of Google’s projects are reproducing high-level quirks of the human visual cortex.
While the work that the visual cortex does is complex and hard to crack (from where we are now), it doesn’t seem like being able to replicate that leads to AGI. Is there a reason I should think otherwise?
There is the ‘one learning algorithm’ hypothesis, that most of the brain uses a single algorithm for learning and pattern recognition. Rather than specialized modules for doing vision, and another for audio, etc.
The evidence experiments where they cut the connection from the eyes to the visual cortex in an animal, and rerouted it to the auditory cortex (and I think vice versa.) The animal then learned to see fine, and it’s auditory cortex just learned how to do vision instead.
which is more than you could say about e.g. symbolic logic; human minds don’t run on logic at all
This seems an odd thing to say. I would say that representation learning (the thing that neural nets do) and compositionality (the thing that symbolic logic does) are likely both part of the puzzle?
The outside view is not very good for predicting technology. Every technology has an eternity of not existing, until suddenly one day it exists out of the blue.
Now no one is saying that deep learning is going to be AGI in 10 years. In fact the deep learning experts have been extremely skeptical of AGI in all forms, and are certainly not promoting that view. But I think it’s a very reasonable opinion that it will lead to AGI within the next few decades. And I believe sooner rather than later.
The reasons that ‘this time it is different’:
NNs are extraordinarily general. I don’t think you can say this about other AI approaches. I mean search and planning algorithms are pretty general. But they fall back on needing heuristics to shrink the search space. And how do you learn heuristics? It goes back to being a machine learning problem. And they are starting to solve it. E.g. a deep neural net predicted Go moves made by experts 54% of the time.
The progress you see is a great deal due to computing power advances. Early AI researchers were working with barely any computing power, and a lot of their work reflects that. That’s not to say we have AGI and are just waiting for computers to get fast enough. But computing power allows researchers to experiment and actually do research.
Empirically they have made significant progress on a number of different AI domains. E.g. vision, speech recognition, natural language processing, and Go. A lot of previous AI approaches might have sounded cool in theory, or worked on a single domain, but they could never point to actual success on loads of different AI problems.
It’s more brain like. I know someone will say that they really aren’t anything like the brain. And that’s true, but at a high level there are very similar principles. Learning networks of features and their connections, as opposed to symbolic approaches.
And if you look at the models that are inspired by the brain like HTM, they are sort of converging on similar algorithms. E.g. they say the important part of the cortex is that it’s very sparse and has lateral inhibition. And you see leading researchers propose very similar ideas.
Whereas the stuff they do differently is mostly because they want to follow biological constraints. Like only local interactions, little memory, only single bits of information at a time. And these aren’t restrictions that real computers have too much, so we don’t necessarily need to copy biology in those respects and can do things differently, and even better.
Several of the above claims don’t seem that true to me.
Statistical methods are also very general. And neural nets definitely need heuristics (LSTMs are basically a really good heuristic for getting NNs to train well).
I’m not aware of great success in Go? 54% accuracy is very hard to interpret in a vaccuum in terms of how impressed to be.
When statistical methods displaced logical methods it’s because they led to lots of progress on lots of domains. In fact, the delta from logical to statistical was probably much larger than the delta from classical statistical learning to neural nets.
I consider deep learning to be in the family of statistical methods. The problem with previous statistical methods is that they were shallow and couldn’t learn very complicated functions or structure. No one ever claimed that linear regression would lead to AGI.
I’m not aware of great success in Go? 54% accuracy is very hard to interpret in a vaccuum in terms of how impressed to be.
That narrows the search space to maybe 2 moves or so per board. Which makes heuristic searching algorithms much more practical. You can not only generate good moves and predict what a human will do, but you can combine that with brute force and search much deeper than a human as well.
And neural nets definitely need heuristics
I mean that NNs learn heuristics. They do require heuristics in the learning algorithm, but not ones that are specific to the domain. Whereas search algorithms depend on lots of domain dependent, manually created heuristics.
Deep Learning is the latest thing in AI. I predict that it will be exactly as successful at achieving AGI as all previous latest things. By which I mean that in 10 years it will be just another chapter in the latest edition of Russell and Norvig.
Purely on Outside View grounds, or based on something more?
Outside View only. That’s the way it’s always worked out before, and I’m not seeing anything specific to Deep Learning to suggest that this time, it will be different. But I am not a professional in this field.
So, some Inside View reasons to think this time might be different:
The results look better, and in particular, some of Google’s projects are reproducing high-level quirks of the human visual cortex.
The methods can absorb far larger amounts of computing power. Previous approaches could not, which makes sense as we didn’t have the computing power for them to absorb at the time, but the human brain does appear to be almost absurdly computation-heavy. Moore’s Law is producing a difference in kind.
That said, I (and most AI researchers, I believe) would agree that deep recurrent networks are only part of the puzzle. The neat thing is, they do appear to be part of the puzzle, which is more than you could say about e.g. symbolic logic; human minds don’t run on logic at all. We’re making progress, and I wouldn’t be surprised if deep learning is part of the first AGI.
While the work that the visual cortex does is complex and hard to crack (from where we are now), it doesn’t seem like being able to replicate that leads to AGI. Is there a reason I should think otherwise?
There is the ‘one learning algorithm’ hypothesis, that most of the brain uses a single algorithm for learning and pattern recognition. Rather than specialized modules for doing vision, and another for audio, etc.
The evidence experiments where they cut the connection from the eyes to the visual cortex in an animal, and rerouted it to the auditory cortex (and I think vice versa.) The animal then learned to see fine, and it’s auditory cortex just learned how to do vision instead.
This seems an odd thing to say. I would say that representation learning (the thing that neural nets do) and compositionality (the thing that symbolic logic does) are likely both part of the puzzle?
The outside view is not very good for predicting technology. Every technology has an eternity of not existing, until suddenly one day it exists out of the blue.
Now no one is saying that deep learning is going to be AGI in 10 years. In fact the deep learning experts have been extremely skeptical of AGI in all forms, and are certainly not promoting that view. But I think it’s a very reasonable opinion that it will lead to AGI within the next few decades. And I believe sooner rather than later.
The reasons that ‘this time it is different’:
NNs are extraordinarily general. I don’t think you can say this about other AI approaches. I mean search and planning algorithms are pretty general. But they fall back on needing heuristics to shrink the search space. And how do you learn heuristics? It goes back to being a machine learning problem. And they are starting to solve it. E.g. a deep neural net predicted Go moves made by experts 54% of the time.
The progress you see is a great deal due to computing power advances. Early AI researchers were working with barely any computing power, and a lot of their work reflects that. That’s not to say we have AGI and are just waiting for computers to get fast enough. But computing power allows researchers to experiment and actually do research.
Empirically they have made significant progress on a number of different AI domains. E.g. vision, speech recognition, natural language processing, and Go. A lot of previous AI approaches might have sounded cool in theory, or worked on a single domain, but they could never point to actual success on loads of different AI problems.
It’s more brain like. I know someone will say that they really aren’t anything like the brain. And that’s true, but at a high level there are very similar principles. Learning networks of features and their connections, as opposed to symbolic approaches.
And if you look at the models that are inspired by the brain like HTM, they are sort of converging on similar algorithms. E.g. they say the important part of the cortex is that it’s very sparse and has lateral inhibition. And you see leading researchers propose very similar ideas.
Whereas the stuff they do differently is mostly because they want to follow biological constraints. Like only local interactions, little memory, only single bits of information at a time. And these aren’t restrictions that real computers have too much, so we don’t necessarily need to copy biology in those respects and can do things differently, and even better.
Several of the above claims don’t seem that true to me.
Statistical methods are also very general. And neural nets definitely need heuristics (LSTMs are basically a really good heuristic for getting NNs to train well).
I’m not aware of great success in Go? 54% accuracy is very hard to interpret in a vaccuum in terms of how impressed to be.
When statistical methods displaced logical methods it’s because they led to lots of progress on lots of domains. In fact, the delta from logical to statistical was probably much larger than the delta from classical statistical learning to neural nets.
I consider deep learning to be in the family of statistical methods. The problem with previous statistical methods is that they were shallow and couldn’t learn very complicated functions or structure. No one ever claimed that linear regression would lead to AGI.
That narrows the search space to maybe 2 moves or so per board. Which makes heuristic searching algorithms much more practical. You can not only generate good moves and predict what a human will do, but you can combine that with brute force and search much deeper than a human as well.
I mean that NNs learn heuristics. They do require heuristics in the learning algorithm, but not ones that are specific to the domain. Whereas search algorithms depend on lots of domain dependent, manually created heuristics.