I agree it’s an attempt to poke Elon, although I suspect he knew that he’d never take the bet. Also agree that anything involving real world robotics in unknown environments is massively more difficult. Having said that, the criteria from Effective Altuirism here:
for any human who can do any job, there is a computer program (not necessarily the same one every time) that can do the same job for $25/hr or less
do say ‘any job’, and we often seem to forget how many jobs require insane levels of dexterity and dealing with the unknown. We could think about the difficulty of building a robot plasterer or car mechanic for example, and see similar levels of complexity, if we pay attention to all the tasks they actually have to do. So I think it fair to have it part of AGI. I do agree that more detailed predictions would be hugely helpful. Marcus’s colleague, Rodney Brooks, has a fun scorecard of predictions for robotics and AI here:
https://rodneybrooks.com/predictions-scorecard-2022-january-01/ which I think is quite useful. As an aside, I had a fun 20 minute chat with GPT-3 today and convinced myself that it doesn’t have the slightest understand of meaning at all! Can send the transcript if interested.
I’d agree with that, I just strongly suspect we can hit dangerous capability without running this experiment first given how research proceeds. If there’s an AI system displaying other blatant signs of being an AGI (by this post’s definition, and assuming non-foom situation, and assuming we’re not dead yet), I won’t bother spending much time wondering about whether it could be a cook.
As an aside, I had a fun 20 minute chat with GPT-3 today and convinced myself that it doesn’t have the slightest understand of meaning at all!
Yup- GPT-3 is shallow in a lot of important ways. It often relies on what appears to be interpolation and memorization. The part that worries me is that architectures like it can still do very difficult reasoning tasks that many humans can’t, like the MATH dataset and minerva. When I look at those accomplishments, I’m not thinking “wow this ML architecture is super duper smart and amazing,” I think “uh oh that part of reasoning is apparently easy if current transformers can do it, while simultaneously failing at trivial things.” We keep getting signals that more and more of our ineffable cognitive skills are… just not that hard.
As we push into architectures that rely more on generalization through explicit reasoning (or maybe even interpolation/memorization at sufficiently absurd scales), a lot of those goofy little mistakes are going to collapse. I’m really worried that an AI that is built for actual reasoning with an architecture able to express what reasoning entails algorithmically is going to be a massive discontinuity, and that it might show up in less than 2 years. It might not take us all the way to AGI in one step, but I’m not looking forward to it.
I really dislike that, as a byproduct of working on safety research, I keep coming up with what look like promising avenues of research for massive capability gain. They seem so much easier to find than good safety ideas, or good ideas in the other fields I work in. I’ve done enough research that I know they wouldn’t all pan out, but the apparent ease is unsettling.
I think you need to be sceptical about what kind of reasoning these systems are actually doing. My contention is that they are all shallow. A system that is trained on near-infinite training sets can look indistinguishable from one that can do deep reasoning, but is in fact just pattern-matching. Or might be. This paper is very pertinent I think: https://arxiv.org/abs/2205.11502 short summary: train a deep network on examples from a logical reasoning task, obtain near-perfect validation error, but find it hasn’t learnt the task at all! It’s learned arbitrary statistical properties of the dataset, completely unrelated to the task. Which is what deep learning does by default. That isn’t going to go away with scale—if anything, it will get worse. And if we say we’ll fix it by adding ‘actual reasoning’, well… good luck! AI spent 2 decades trying to build symbolic reasoning systems, getting that to work is incredibly hard. Now I haven’t actually read up on the Minerva results yet, and will do so, but I do think we need to exercise caution before attributing reasoning to something, if there are dumber ways to get the same behaviour. To me all this says is that we need a new paradigm entirely to get anywhere close to AGI. That’s not impossible, but it makes me sufficiently confident that it’s going to be decades, if not a couple of centuries.
My contention is that they are all shallow. A system that is trained on near-infinite training sets can look indistinguishable from one that can do deep reasoning, but is in fact just pattern-matching.
I agree.
This is a big part of what my post is about.
We have AI that is obviously dumb, in the sense of failing on trivial tasks and having mathematically provable strict bounds.
That type of AI is eating progressively larger chunks of things we used to call “intelligence.”
The things we used to call intelligence are, apparently, easy.
We should expect (and have good reason to believe) that more of what we currently call intelligence to be easy, and it may very well be consumed by dumb architectures.
Less dumb architectures are being worked on, and do not require paradigm shifts.
Uh oh.
This is a statement mostly about the problem, not the problem solver. The problem we thought was hard just isn’t.
And if we say we’ll fix it by adding ‘actual reasoning’, well… good luck! AI spent 2 decades trying to build symbolic reasoning systems, getting that to work is incredibly hard.
Going to be deliberately light on details here again, sorry. When I say ‘actual reasoning,’ I mean AI that is trained in a way that learning the capabilities provided by reasoning is a more direct byproduct, rather than a highly indirect feature that arises from its advantages in blind token prediction. (Though a sufficiently large dumb system might manage to capture way too much anyway.)
I’m not suggesting we need a new SHRDLU. There are paths fully contained within the current deep learning paradigm. There is empirical support for this.
That’s a very well-argued point. I have precisely the opposite intuition of course, but I can’t deny the strength of your argument.. I tend to be less interested in tasks that are well-bounded, than those that are open-ended and uncertain. I agree that much of what we call intelligent might be much simpler. But then I think common sense reasoning is much harder. I think maybe I’ll try to draw up my own list of tasks for AGI :)
Is this research into ‘actual reasoning’ that you’re deliberately being light on details about something that is out in the public (e.g. on arxiv), or is this something you’ve witnessed privately and anticipate will become public in the near future?
Most of it is the latter, but to be clear, I do not have inside information about what any large organization is doing privately, nor have I seen an “oh no we’re doomed” proof of concept. Just some very obvious “yup that’ll work” stuff. I expect adjacent things to be published at some point soonishly just because the ideas are so simple and easily found/implemented independently. Someone might have already and I’m just not aware of it. I just don’t want to be the one to oops and push on the wrong side of the capability-safety balance.
I agree it’s an attempt to poke Elon, although I suspect he knew that he’d never take the bet. Also agree that anything involving real world robotics in unknown environments is massively more difficult. Having said that, the criteria from Effective Altuirism here:
for any human who can do any job, there is a computer program (not necessarily the same one every time) that can do the same job for $25/hr or less
do say ‘any job’, and we often seem to forget how many jobs require insane levels of dexterity and dealing with the unknown. We could think about the difficulty of building a robot plasterer or car mechanic for example, and see similar levels of complexity, if we pay attention to all the tasks they actually have to do. So I think it fair to have it part of AGI. I do agree that more detailed predictions would be hugely helpful. Marcus’s colleague, Rodney Brooks, has a fun scorecard of predictions for robotics and AI here:
https://rodneybrooks.com/predictions-scorecard-2022-january-01/
which I think is quite useful. As an aside, I had a fun 20 minute chat with GPT-3 today and convinced myself that it doesn’t have the slightest understand of meaning at all! Can send the transcript if interested.
I’d agree with that, I just strongly suspect we can hit dangerous capability without running this experiment first given how research proceeds. If there’s an AI system displaying other blatant signs of being an AGI (by this post’s definition, and assuming non-foom situation, and assuming we’re not dead yet), I won’t bother spending much time wondering about whether it could be a cook.
Yup- GPT-3 is shallow in a lot of important ways. It often relies on what appears to be interpolation and memorization. The part that worries me is that architectures like it can still do very difficult reasoning tasks that many humans can’t, like the MATH dataset and minerva. When I look at those accomplishments, I’m not thinking “wow this ML architecture is super duper smart and amazing,” I think “uh oh that part of reasoning is apparently easy if current transformers can do it, while simultaneously failing at trivial things.” We keep getting signals that more and more of our ineffable cognitive skills are… just not that hard.
As we push into architectures that rely more on generalization through explicit reasoning (or maybe even interpolation/memorization at sufficiently absurd scales), a lot of those goofy little mistakes are going to collapse. I’m really worried that an AI that is built for actual reasoning with an architecture able to express what reasoning entails algorithmically is going to be a massive discontinuity, and that it might show up in less than 2 years. It might not take us all the way to AGI in one step, but I’m not looking forward to it.
I really dislike that, as a byproduct of working on safety research, I keep coming up with what look like promising avenues of research for massive capability gain. They seem so much easier to find than good safety ideas, or good ideas in the other fields I work in. I’ve done enough research that I know they wouldn’t all pan out, but the apparent ease is unsettling.
I think you need to be sceptical about what kind of reasoning these systems are actually doing. My contention is that they are all shallow. A system that is trained on near-infinite training sets can look indistinguishable from one that can do deep reasoning, but is in fact just pattern-matching. Or might be. This paper is very pertinent I think:
https://arxiv.org/abs/2205.11502
short summary: train a deep network on examples from a logical reasoning task, obtain near-perfect validation error, but find it hasn’t learnt the task at all! It’s learned arbitrary statistical properties of the dataset, completely unrelated to the task. Which is what deep learning does by default. That isn’t going to go away with scale—if anything, it will get worse. And if we say we’ll fix it by adding ‘actual reasoning’, well… good luck! AI spent 2 decades trying to build symbolic reasoning systems, getting that to work is incredibly hard.
Now I haven’t actually read up on the Minerva results yet, and will do so, but I do think we need to exercise caution before attributing reasoning to something, if there are dumber ways to get the same behaviour.
To me all this says is that we need a new paradigm entirely to get anywhere close to AGI. That’s not impossible, but it makes me sufficiently confident that it’s going to be decades, if not a couple of centuries.
I agree.
This is a big part of what my post is about.
We have AI that is obviously dumb, in the sense of failing on trivial tasks and having mathematically provable strict bounds.
That type of AI is eating progressively larger chunks of things we used to call “intelligence.”
The things we used to call intelligence are, apparently, easy.
We should expect (and have good reason to believe) that more of what we currently call intelligence to be easy, and it may very well be consumed by dumb architectures.
Less dumb architectures are being worked on, and do not require paradigm shifts.
Uh oh.
This is a statement mostly about the problem, not the problem solver. The problem we thought was hard just isn’t.
Going to be deliberately light on details here again, sorry. When I say ‘actual reasoning,’ I mean AI that is trained in a way that learning the capabilities provided by reasoning is a more direct byproduct, rather than a highly indirect feature that arises from its advantages in blind token prediction. (Though a sufficiently large dumb system might manage to capture way too much anyway.)
I’m not suggesting we need a new SHRDLU. There are paths fully contained within the current deep learning paradigm. There is empirical support for this.
That’s a very well-argued point. I have precisely the opposite intuition of course, but I can’t deny the strength of your argument.. I tend to be less interested in tasks that are well-bounded, than those that are open-ended and uncertain. I agree that much of what we call intelligent might be much simpler. But then I think common sense reasoning is much harder. I think maybe I’ll try to draw up my own list of tasks for AGI :)
Is this research into ‘actual reasoning’ that you’re deliberately being light on details about something that is out in the public (e.g. on arxiv), or is this something you’ve witnessed privately and anticipate will become public in the near future?
Here is a paper from January 2022 on arXiv that details the sort of generalization-hop we’re seeing models doing.
Most of it is the latter, but to be clear, I do not have inside information about what any large organization is doing privately, nor have I seen an “oh no we’re doomed” proof of concept. Just some very obvious “yup that’ll work” stuff. I expect adjacent things to be published at some point soonishly just because the ideas are so simple and easily found/implemented independently. Someone might have already and I’m just not aware of it. I just don’t want to be the one to oops and push on the wrong side of the capability-safety balance.