Hmm, mostly just articles where they get better results with more NN layers/more examples, which are both limited by hardware capacity and have seen large gains from things like using GPUs. Current algos still have far fewer “neurons” than the actual brain AFAIK. Plus, in general, faster hardware allows for faster/cheaper experimentation with different algorithms.
I’ve seen some AI researchers (eg Yann Lecun on Facebook) emphasizing that fundamental techniques haven’t changed that much in decades, yet results continue to improve with more computation.
Current algos still have far fewer “neurons” than the actual brain AFAIK.
This is not primarily because of limitations in computing power. The relevant limitation is on the complexity of the model you can train, without overfitting, in comparison to the volume of data you have (a larger data set permits a more complex model).
Besides what fezziwig said, which is correct, the other issue is the fundamental capabilities of the domain you are looking at. I figured something like this was the source of the error, which is why I asked for context.
Neural networks, deep or otherwise, are basically just classifiers. The reason we’ve seen large advancements recently in machine learning is chiefly because of the immense volumes of data available to these classifier-learning programs. Machine learning is particularly good at taking heaps of structured or unstructured data and finding clusters, then coming up with ways to classify new data into one of those identified clusters. The more data you have, the most detail that can be identified, and the better your classifiers become. Certainly you need a lot of hardware to process the mind boggling amounts of data that are being pushed through these machine learning tools, but hardware is not the limiter, available data is. Giant companies like Google and Facebook are building better and better classifiers not because they have more hardware available, but because they have more data available (chiefly because we are choosing to escrow our personal lives to these companies servers, but that’s an aside).
In as much as machine learning tends to dominate current approaches to narrow AI, you could be excused for saying “the biggest limitation on AI development is availabilities of data.” But you mentioned safety, and AI safety around here is a codeword for general AI, and general AI is truly a software problem that has very little to do with neural networks, data availability, or hardware speeds. “But human brains are networks of neurons!” you reply. True. But the field of computer algorithms called neural networks is a total misnomer. A “neural network” is an algorithm inspired by an over simplification of a misconception of how brains worked that dates back to the 1950′s / 1960′s.
Developing algorithms that are actually capable of performing general intelligence tasks, either bio-inspired or de novo, is the field of artificial general intelligence. And that field is currently software limited. We suspect we have the computational capability to run a human-level AGI today, if only we had the know-how to write one.
I already know all this (from a combination of intro-to-ML course and reading writing along the same lines by Yann Lecun and Andrew Ng), and I’m still leaning towards hardware being the limiting factor (ie I currently don’t think your last sentence is true).
I think you have the right idea, but it’s a mistake to conflate “needs a big corpus of data” and “needs lots of hardware”. Hardware helps, the faster the training goes the more experiments you can do, but a lot of the time the gating factor is the corpus itself.
For example, if you’re trying to train a neural net to solve the “does this photo contain a bird?” problem, you need a bunch of photos which vary at random on the bird/not-bird axis, and you need human raters to go through and tag each photo as bird/not-bird. There are many ways to lose here. For example, your variable of interest might be correlated to something boring (maybe all the bird photos were taken in the morning, and all the not-bird photos were taken in the afternoon), or your raters have to spend a lot of time with each photo (imagine you want to do beak detection, instead of just bird/not-bird: then your raters have to attach a bunch of metadata to each training image, describing the beak position in each bird photo).
The difference between hardware that’s fast enough to fit many iterations into a time span suitable for writing a paper vs. hardware that is slow enough that feedback is infrequent seems fairly relevant to how fast the software can progress.
New insights depend crucially on feedback gotten from trying out the old insights.
Hmm, mostly just articles where they get better results with more NN layers/more examples, which are both limited by hardware capacity and have seen large gains from things like using GPUs. Current algos still have far fewer “neurons” than the actual brain AFAIK. Plus, in general, faster hardware allows for faster/cheaper experimentation with different algorithms.
I’ve seen some AI researchers (eg Yann Lecun on Facebook) emphasizing that fundamental techniques haven’t changed that much in decades, yet results continue to improve with more computation.
This is not primarily because of limitations in computing power. The relevant limitation is on the complexity of the model you can train, without overfitting, in comparison to the volume of data you have (a larger data set permits a more complex model).
Besides what fezziwig said, which is correct, the other issue is the fundamental capabilities of the domain you are looking at. I figured something like this was the source of the error, which is why I asked for context.
Neural networks, deep or otherwise, are basically just classifiers. The reason we’ve seen large advancements recently in machine learning is chiefly because of the immense volumes of data available to these classifier-learning programs. Machine learning is particularly good at taking heaps of structured or unstructured data and finding clusters, then coming up with ways to classify new data into one of those identified clusters. The more data you have, the most detail that can be identified, and the better your classifiers become. Certainly you need a lot of hardware to process the mind boggling amounts of data that are being pushed through these machine learning tools, but hardware is not the limiter, available data is. Giant companies like Google and Facebook are building better and better classifiers not because they have more hardware available, but because they have more data available (chiefly because we are choosing to escrow our personal lives to these companies servers, but that’s an aside).
In as much as machine learning tends to dominate current approaches to narrow AI, you could be excused for saying “the biggest limitation on AI development is availabilities of data.” But you mentioned safety, and AI safety around here is a codeword for general AI, and general AI is truly a software problem that has very little to do with neural networks, data availability, or hardware speeds. “But human brains are networks of neurons!” you reply. True. But the field of computer algorithms called neural networks is a total misnomer. A “neural network” is an algorithm inspired by an over simplification of a misconception of how brains worked that dates back to the 1950′s / 1960′s.
Developing algorithms that are actually capable of performing general intelligence tasks, either bio-inspired or de novo, is the field of artificial general intelligence. And that field is currently software limited. We suspect we have the computational capability to run a human-level AGI today, if only we had the know-how to write one.
I already know all this (from a combination of intro-to-ML course and reading writing along the same lines by Yann Lecun and Andrew Ng), and I’m still leaning towards hardware being the limiting factor (ie I currently don’t think your last sentence is true).
I think you have the right idea, but it’s a mistake to conflate “needs a big corpus of data” and “needs lots of hardware”. Hardware helps, the faster the training goes the more experiments you can do, but a lot of the time the gating factor is the corpus itself.
For example, if you’re trying to train a neural net to solve the “does this photo contain a bird?” problem, you need a bunch of photos which vary at random on the bird/not-bird axis, and you need human raters to go through and tag each photo as bird/not-bird. There are many ways to lose here. For example, your variable of interest might be correlated to something boring (maybe all the bird photos were taken in the morning, and all the not-bird photos were taken in the afternoon), or your raters have to spend a lot of time with each photo (imagine you want to do beak detection, instead of just bird/not-bird: then your raters have to attach a bunch of metadata to each training image, describing the beak position in each bird photo).
The difference between hardware that’s fast enough to fit many iterations into a time span suitable for writing a paper vs. hardware that is slow enough that feedback is infrequent seems fairly relevant to how fast the software can progress.
New insights depend crucially on feedback gotten from trying out the old insights.