The problem that I have is that Eliezer, along with MIRI, and many other rationalists automatically assume that the eventuality of artificial intelligence exceeding human intelligence equates with that occurring with great speed. What evidence do we have that AI will suddenly and radically exceed human capabilities in a generalized fashion in a short period of time? The AI advancements pointed to in the piece, were all “narrow AIs”, which progressed past human capabilities after a significant investment of time, research effort and computational hardware. What, beyond some nameless fear, is causing Eliezer to say that AI will suddenly progress in a generalized fashion across all fronts, when everything until now has been progress along fairly narrow fronts?
On a broader level, I see an assumption constantly made that an AGI system will be a single system. What evidence do we have of AI being a single integrated system rather than multiple specialized systems, each of which do a single thing better than all humans, but none of which do everything better?
People aren’t assuming that AI exceeding human intelligence “equates with that occurring with great speed”; they’re arguing for the latter point separately. E.g., see:
Or, for a much quicker and more impressionistic argument, this post on FB.
Another simple argument: “Human cognition didn’t evolve to do biochemistry, nuclear engineering, or computer science; those capabilities just ‘came for free’ with the very different set of cognitive problems human brains evolved to solve in our environment of evolutionary adaptedness. This suggests that there’s such a thing as ‘general intelligence’ in the sense that there’s a kind of reasoning that lets you learn all those sciences without needing an engineer to specially design new brains or new brain modules for each new domain; and it’s the kind of capacity that a blind engineering process like natural selection was able to stumble on while ‘trying’ to solve a very different set of problems.”
Some other threads that bear directly on this question include:
What’s the track record within AI, or in automation in general? When engineers try to outperform biology on some specific task (and especially on cognitive tasks), how often do they hit a wall at par-biology performance; and when they don’t hit a wall, how often do they quickly shoot past biological performance on the intended dimension?
Are humans likely to be near an intelligence ceiling, or near a point where evolution was hitting diminishing returns (for reasons that generalize to AI)?
How hardware-intensive is AGI likely to be? How does this vary for, e.g., 10-year versus 30-year timelines?
Along how many dimensions might AGI improve on human intelligence? How likely is it that early AGI systems will be able to realize some of these improvements, and to what degree; and how easy is it likely to be to leverage easier advantages to achieve harder ones?
How tractable is technological progress (of the kind we might use AGI to automate) in general? More broadly, if you have (e.g.) AGI systems that can do the very rough equivalent of 1000 serial years of cognitive work by 10 collaborating human scientists over the span of a couple of years, how much progress can those systems make on consequential real-world problems?
If large rapid capability gains are available, how likely is it that actors will be willing (and able) to go slow? Instrumental convergence and Gwern’s post on tool AIs are relevant here.
Each of these is a big topic in its own right. I’m noting all these different threads because I want to be clear about how many different directions you can go in if you’re curious about this; obviously feel free to pick just one thread and start the discussion there, though, since all of this can be a lot to try to cover simultaneously, and it’s useful to ask questions and start hashing things out before you’ve read literally everything that’s been written on the topic.
On the same topic, see also my paper How Feasible is the Rapid Development of Artificial Superintelligence (recently accepted for publication in the 21st Century Frontiers focus issue of Physica Scripta), in which I argue that the things that we know about human expertise and intelligence seem to suggest that the process of scaling up from human-level intelligence to superhuman qualitative intelligence might be relatively fast and simple.
How tractable is technological progress (of the kind we might use AGI to automate) in general? More broadly, if you have (e.g.) AGI systems that can do the very rough equivalent of 1000 serial years of cognitive work by 10 collaborating human scientists over the span of a couple of years, how much progress can those systems make on consequential real-world problems?
How much science is cognitive work vs running an experiment in the real world? Have there been attempts to quantify that?
MIRI and other people thinking about strategies for ending the risk period use “how much physical experimentation is needed, how fast can the experiments be run, how much can they be parallelized, how hard is it to build and operate the equipment, etc.?” as one of the key criteria for evaluating strategies. The details depend on what technologies you think are most likely to be useful for addressing existential risk with AGI (which is not completely clear, though there are plausible ideas out there). We expect a lot of speed advantages from AGI, so the time cost of experiments is an important limiting factor.
Are there any organisations set up to research this kind of question (going into universities and studying research)? I’m wondering if we need a specialism called something like AI prediction, which aims to get this kind of data.
If this topic interests you, you may want to reach out to the Open Philanthropy Project, as they’re interested in supporting efforts to investigate these questions in a more serious way.
Hi Rob, I had hoped to find people I could support. I am interested in the question. I’ll see if I thnk it is more important than the other questions I am interested in.
They look like they are set up for researching existing literature and doing surveys, but they are not necessarily set up to do studies that collect data in labs.
The project is provisionally organized as a collection of posts concerning particular issues or bodies of evidence, describing what is known and attempting to synthesize a reasonable view in light of available evidence.
Are there any reasons for this expectation? In software development generally and machine learning specifically it often takes much longer to solve a problem for the first time than successive instances. The intuition this primes is that a proto-AGI is likely to stumble and require manual assistance a lot the first time it attempts any one Thing, and generally the Thing will take longer to do with an AI than without. The advantage of course is that afterwards similar problems are solved quickly and efficiently, which is what makes working on AI pay off.
AFAICT, the claim that any form of not-yet-superhuman AGI will quickly, efficiently, and autonomously solve the problems it encounters in solving more and more general classes of problems (aka “FOOM”) is entirely ungrounded.
Dawkins’s “Middle World” idea seems relevant here. We live in Middle World, but we investigate phenomena across a wide range of scales in space and time. It would at least be a little surprising to discover that the pace at which we do it is special and hard to improve on.
I agree that research can probably be improved upon quickly and easily. Lab on a chip is obviously a way we are doing that currently. If an AGI system has got the backing of a large company or country and can get new things fabbed in secrecy it can improve on these kinds of things. But I still think it is worth trying to quantify things. We can get ideas about stealth scenarios where the AGI is being developed by non-state/megacorp actors that can’t easily fab new things. We can also get ideas about how useful things like lab on a chip are for speeding up the relevant science. Are we out of low hanging fruit and is it taking us more effort to find novel interesting chemicals/materials?
I note that this had been downvoted into negative karma and want to push back on that. A lot of us feel like You Should Know This Already but many don’t, many others disagree, and it’s a very important question that the OP skips.
This feels like an excellent place for someone to ask for the evidence, and Rob does a good job doing that, and I don’t want people to feel punished for asking such questions nor do I want the question and Rob’s answer to become hidden.
If anything, it seems odd that our front page resources don’t currently include an easy pointing towards such evidence.
This feels like an excellent place for someone to ask for the evidence
Was that really what the grandparent comment was doing, though? The impression I got was that the original commenter was simply using the question as a rhetorical device in order to reinforce the (false) impression that MIRI et al. “automatically assume” things relating to the growth curve of superintelligent AI, and that kind of rhetoric is certainly not something I want to encourage.
I think Quanticle should have asked questions, rather than making strong claims like “MIRI automatically assumes p” without looking into the issue more. On the other hand, I’m glad that someone raised issues like these in this comments section (given that disagreements and misunderstandings on these issues are pretty common), and I care more about the issues getting discussed and about people sharing their current epistemic state than about punishing people who rushed to a wrong conclusion or had tone issues or whatever. (If I were the Emperor of Karma I might allocate something like ‘net +2 karma’ to mildly reward this level of openness and directness, without over-rewarding lack-of-scholarship etc.)
Re ‘maybe this was all a ploy / rhetorical device’, I’m skeptical that that’s true in any strong/unusual sense. I also want to discourage treating it as normal, at the outset of a debate over some set of factual issues, to publicly speculate that the person on the other side has bad motives (in an accusatory/critical/dismissive/etc. way). There may be extreme cases where we’re forced to do that at the outset of a factual debate, but it should be pretty rare, given what it can do to discussion when those sorts of accusations are commonplace.
Re ‘maybe this was all a ploy / rhetorical device’, I’m skeptical that that’s true in any strong/unusual sense.
I don’t think that there’s anything particularly unusual about someone asking “Is there any evidence for claim X?” to imply that, no, there is not enough evidence for claim X. Rhetorical questions are such a common argumentative technique that you can sometimes employ them without even being consciously aware of it. That still doesn’t make it the kind of style of discourse I approve of, however, and downvoting is a compact way of expressing that disapproval.
I also want to discourage treating it as normal, at the outset of a debate over some set of factual issues, to publicly speculate that the person on the other side has bad motives (in an accusatory/critical/dismissive/etc. way).
To be clear, I didn’t reply to the original comment at all; my initial comment upthread was written purely in response to Zvi’s allegation that the downvoters of Quanticle’s comment were being unfair and/or petty. I disagreed with the notion that there was no valid reason to downvote, and I replied for that reason and that reason only. I certainly didn’t intend my comment to be interpreted as “public speculation” regarding Quanticle’s motives, only as an indication that the phrasing used could give the impression of bad motives, which I think is just as important as whether it was actually intended that way. More generally, however:
You said that the substance of a comment is more important than its tone, and I certainly don’t disagree, but that still doesn’t mean that issues relating to tone are unimportant. In fact, I’d go so far as to say that the way a commenter phrases certain things can very strongly shape the subsequent flow of discussion, and that in some cases the effects are strong enough to outweigh the actual substance of their comment entirely, especially when there’s little to no substance to begin with (as in this case). Given that, I think voting based on such “ephemeral” considerations as tone and phrasing is just as valid as any other means of voting, and I take issue with the idea that you can’t downvote and/or criticize someone for anything other than the purely denotative meaning of their statements.
How many people downvoted because Grr I Disagree and How Dare They Question EY, and how many because they didn’t like how the poster leaped from “I haven’t seen any reason to think this” to “they must be automatically assuming this”? This wasn’t so much asking for evidence as… assuming there was none. That tends to annoy people, and I think it makes sense on a productive-discourse level to discourage such posts. On a teach-the-newcomer-and-don’t-drive-them-out-for-not-being-perfectly-rational-yet level, however, I think you have a point.
As an intuition pump, imagine that to become “fully human”, whatever that means, the AI needs to have three traits, let’s call them “A”, “B”, and “C”, whatever those could be. It seems unlikely that the AI would gain all those three traits in the same version. More likely, first we will get an AI 1.0 that has the trait “A”, but lacks traits “B” and “C”. At some later time, we will have an AI 2.0 that has traits “A” and “B”, but still lacks “C”.
Now, the important thing is that if the AI 1.0 had the trait “A” at human level, the scientific progress at “A” probably still continued, so the AI 2.0 most likely already has the trait “A” at super-human level. So it has super-human “A”, human-level “B”, but still lacks “C”.
And for the same reason, when later we have an AI 3.0 that finally has all the traits “A”, “B”, and “C” at least at the human level, it will likely already have traits “A” and “B” at super-human level; the trait “A” probably at insanely-super-human level. In other words, the first “fully human” AI will actually already be super-human in some aspects, simply because it is unlikely that all aspects would reach the human level at the same time.
For example, the first “fully human” AI will easily win the chess tournaments, simply because current AIs can already win them. Instead of some average Joe, the first “fully human” AI will be, at least, some kind of super-Kasparov.
How dangerous it is to have some super-human traits, while being “fully human” otherwise? Depends on the trait. Notice how smarter can people act when you simply give them more time, or better memory (e.g. having a paper, or some personal wiki software), or ability to work in groups. If the AI is otherwise like a human, except e.g. 100 times faster, or able to keep a wiki in its head, or able to do real multitasking (i.e. split into a few independent processes, and explore the issue from multiple angles at the same time), that could already make it quite smart. (Imagine how your life would change if at any moment you could magically take an extra hour to think about stuff, having all relevant books in your head, and an invisible team of equally smart discussion partners.) And these are just quite boring examples of human traits; it could also be e.g. 100 times greater ability to recognize patterns; or perhaps some trait we don’t have a word for yet.
Generally, the idea is that “an average human” is a tiny dot on the intelligence scale. If you make small jumps upwards on the scale, the chance that one of your jumps will end exactly at this dot is very small. More likely, your jumps will repeatedly end below this dot, until at some moment the following jump will take over this dot, at some higher place. Humans have many “hardware” limitations that keep them in a narrow interval, such as brains made of meat working at frequency 200 Hz, or heads small enough to allow childbirth. The AI will have none of these. It will have its own hardware limits, but those will follow different rules. So it seems possible that e.g. the AI built in 2022 will work at human equivalent of 20 Hz, and the AI built in 2023 will work at human equivalent of 2000 Hz, simply because someone invented a smart algorithm allowing much faster simulation of neurons, or used a network of thousand computers instead of one supercomputer, or coded the critical parts in C++ instead of Lisp and had them run on GPU, etc.
But perhaps it is enough to accept that the first “fully human” AI could beat you at chess even in its sleep, and ask yourself what is the chance that chess would the only such example.
“For example, the first “fully human” AI will easily win the chess tournaments, simply because current AIs can already win them.” No they can’t. *Chess playing programs* can easily win tournaments, self-driving cars and sentiment analysers can’t. An AGI that had the ability to run a chess playing program would be able to win, but the same applies to humans with the same ability.
+1 with this disagreement. ML methods seem to indicate that it’s not quite that simple—they don’t soar beyond human level just from self-improvement, when pointed at themselves they self-improve but relatively very slowly compared to the predictions. I do think it’s possible, but it no longer seems like an obvious thing the way it may have before ML became a real thing.
With respect to being a single system: it’s consistently the case that end-to-end learned neural networks are better at their jobs than plugging together disparately trained networks, which are in turn better at their jobs than neural networks that can only communicate via one-hot vectors (eg, agent populations communicating in english, or ai systems like the heavily multimodal alexa bots). The latter can be much easier to make, but in turn top out much earlier.
The problem that I have is that Eliezer, along with MIRI, and many other rationalists automatically assume that the eventuality of artificial intelligence exceeding human intelligence equates with that occurring with great speed. What evidence do we have that AI will suddenly and radically exceed human capabilities in a generalized fashion in a short period of time? The AI advancements pointed to in the piece, were all “narrow AIs”, which progressed past human capabilities after a significant investment of time, research effort and computational hardware. What, beyond some nameless fear, is causing Eliezer to say that AI will suddenly progress in a generalized fashion across all fronts, when everything until now has been progress along fairly narrow fronts?
On a broader level, I see an assumption constantly made that an AGI system will be a single system. What evidence do we have of AI being a single integrated system rather than multiple specialized systems, each of which do a single thing better than all humans, but none of which do everything better?
People aren’t assuming that AI exceeding human intelligence “equates with that occurring with great speed”; they’re arguing for the latter point separately. E.g., see:
Yudkowsky’s Intelligence Explosion Microeconomics
Hanson and Yudkowsky’s AI-Foom Debate
Bostrom’s Superintelligence
Or, for a much quicker and more impressionistic argument, this post on FB.
Another simple argument: “Human cognition didn’t evolve to do biochemistry, nuclear engineering, or computer science; those capabilities just ‘came for free’ with the very different set of cognitive problems human brains evolved to solve in our environment of evolutionary adaptedness. This suggests that there’s such a thing as ‘general intelligence’ in the sense that there’s a kind of reasoning that lets you learn all those sciences without needing an engineer to specially design new brains or new brain modules for each new domain; and it’s the kind of capacity that a blind engineering process like natural selection was able to stumble on while ‘trying’ to solve a very different set of problems.”
Some other threads that bear directly on this question include:
What’s the track record within AI, or in automation in general? When engineers try to outperform biology on some specific task (and especially on cognitive tasks), how often do they hit a wall at par-biology performance; and when they don’t hit a wall, how often do they quickly shoot past biological performance on the intended dimension?
Are humans likely to be near an intelligence ceiling, or near a point where evolution was hitting diminishing returns (for reasons that generalize to AI)?
How hardware-intensive is AGI likely to be? How does this vary for, e.g., 10-year versus 30-year timelines?
Along how many dimensions might AGI improve on human intelligence? How likely is it that early AGI systems will be able to realize some of these improvements, and to what degree; and how easy is it likely to be to leverage easier advantages to achieve harder ones?
How tractable is technological progress (of the kind we might use AGI to automate) in general? More broadly, if you have (e.g.) AGI systems that can do the very rough equivalent of 1000 serial years of cognitive work by 10 collaborating human scientists over the span of a couple of years, how much progress can those systems make on consequential real-world problems?
If large rapid capability gains are available, how likely is it that actors will be willing (and able) to go slow? Instrumental convergence and Gwern’s post on tool AIs are relevant here.
Each of these is a big topic in its own right. I’m noting all these different threads because I want to be clear about how many different directions you can go in if you’re curious about this; obviously feel free to pick just one thread and start the discussion there, though, since all of this can be a lot to try to cover simultaneously, and it’s useful to ask questions and start hashing things out before you’ve read literally everything that’s been written on the topic.
On the same topic, see also my paper How Feasible is the Rapid Development of Artificial Superintelligence (recently accepted for publication in the 21st Century Frontiers focus issue of Physica Scripta), in which I argue that the things that we know about human expertise and intelligence seem to suggest that the process of scaling up from human-level intelligence to superhuman qualitative intelligence might be relatively fast and simple.
How much science is cognitive work vs running an experiment in the real world? Have there been attempts to quantify that?
MIRI and other people thinking about strategies for ending the risk period use “how much physical experimentation is needed, how fast can the experiments be run, how much can they be parallelized, how hard is it to build and operate the equipment, etc.?” as one of the key criteria for evaluating strategies. The details depend on what technologies you think are most likely to be useful for addressing existential risk with AGI (which is not completely clear, though there are plausible ideas out there). We expect a lot of speed advantages from AGI, so the time cost of experiments is an important limiting factor.
Are there any organisations set up to research this kind of question (going into universities and studying research)? I’m wondering if we need a specialism called something like AI prediction, which aims to get this kind of data.
If this topic interests you, you may want to reach out to the Open Philanthropy Project, as they’re interested in supporting efforts to investigate these questions in a more serious way.
Hi Rob, I had hoped to find people I could support. I am interested in the question. I’ll see if I thnk it is more important than the other questions I am interested in.
AI impacts has done some research in this area, I think.
They look like they are set up for researching existing literature and doing surveys, but they are not necessarily set up to do studies that collect data in labs.
They are still part of the orient step, rather than the observation step.
But still lots of interesting things. Thanks for pointing me at them.
Are there any reasons for this expectation? In software development generally and machine learning specifically it often takes much longer to solve a problem for the first time than successive instances. The intuition this primes is that a proto-AGI is likely to stumble and require manual assistance a lot the first time it attempts any one Thing, and generally the Thing will take longer to do with an AI than without. The advantage of course is that afterwards similar problems are solved quickly and efficiently, which is what makes working on AI pay off.
AFAICT, the claim that any form of not-yet-superhuman AGI will quickly, efficiently, and autonomously solve the problems it encounters in solving more and more general classes of problems (aka “FOOM”) is entirely ungrounded.
Dawkins’s “Middle World” idea seems relevant here. We live in Middle World, but we investigate phenomena across a wide range of scales in space and time. It would at least be a little surprising to discover that the pace at which we do it is special and hard to improve on.
I agree that research can probably be improved upon quickly and easily. Lab on a chip is obviously a way we are doing that currently. If an AGI system has got the backing of a large company or country and can get new things fabbed in secrecy it can improve on these kinds of things.
But I still think it is worth trying to quantify things. We can get ideas about stealth scenarios where the AGI is being developed by non-state/megacorp actors that can’t easily fab new things. We can also get ideas about how useful things like lab on a chip are for speeding up the relevant science. Are we out of low hanging fruit and is it taking us more effort to find novel interesting chemicals/materials?
I note that this had been downvoted into negative karma and want to push back on that. A lot of us feel like You Should Know This Already but many don’t, many others disagree, and it’s a very important question that the OP skips.
This feels like an excellent place for someone to ask for the evidence, and Rob does a good job doing that, and I don’t want people to feel punished for asking such questions nor do I want the question and Rob’s answer to become hidden.
If anything, it seems odd that our front page resources don’t currently include an easy pointing towards such evidence.
Was that really what the grandparent comment was doing, though? The impression I got was that the original commenter was simply using the question as a rhetorical device in order to reinforce the (false) impression that MIRI et al. “automatically assume” things relating to the growth curve of superintelligent AI, and that kind of rhetoric is certainly not something I want to encourage.
I think Quanticle should have asked questions, rather than making strong claims like “MIRI automatically assumes p” without looking into the issue more. On the other hand, I’m glad that someone raised issues like these in this comments section (given that disagreements and misunderstandings on these issues are pretty common), and I care more about the issues getting discussed and about people sharing their current epistemic state than about punishing people who rushed to a wrong conclusion or had tone issues or whatever. (If I were the Emperor of Karma I might allocate something like ‘net +2 karma’ to mildly reward this level of openness and directness, without over-rewarding lack-of-scholarship etc.)
Re ‘maybe this was all a ploy / rhetorical device’, I’m skeptical that that’s true in any strong/unusual sense. I also want to discourage treating it as normal, at the outset of a debate over some set of factual issues, to publicly speculate that the person on the other side has bad motives (in an accusatory/critical/dismissive/etc. way). There may be extreme cases where we’re forced to do that at the outset of a factual debate, but it should be pretty rare, given what it can do to discussion when those sorts of accusations are commonplace.
I don’t think that there’s anything particularly unusual about someone asking “Is there any evidence for claim X?” to imply that, no, there is not enough evidence for claim X. Rhetorical questions are such a common argumentative technique that you can sometimes employ them without even being consciously aware of it. That still doesn’t make it the kind of style of discourse I approve of, however, and downvoting is a compact way of expressing that disapproval.
To be clear, I didn’t reply to the original comment at all; my initial comment upthread was written purely in response to Zvi’s allegation that the downvoters of Quanticle’s comment were being unfair and/or petty. I disagreed with the notion that there was no valid reason to downvote, and I replied for that reason and that reason only. I certainly didn’t intend my comment to be interpreted as “public speculation” regarding Quanticle’s motives, only as an indication that the phrasing used could give the impression of bad motives, which I think is just as important as whether it was actually intended that way. More generally, however:
You said that the substance of a comment is more important than its tone, and I certainly don’t disagree, but that still doesn’t mean that issues relating to tone are unimportant. In fact, I’d go so far as to say that the way a commenter phrases certain things can very strongly shape the subsequent flow of discussion, and that in some cases the effects are strong enough to outweigh the actual substance of their comment entirely, especially when there’s little to no substance to begin with (as in this case). Given that, I think voting based on such “ephemeral” considerations as tone and phrasing is just as valid as any other means of voting, and I take issue with the idea that you can’t downvote and/or criticize someone for anything other than the purely denotative meaning of their statements.
Working on it :-)
How many people downvoted because Grr I Disagree and How Dare They Question EY, and how many because they didn’t like how the poster leaped from “I haven’t seen any reason to think this” to “they must be automatically assuming this”? This wasn’t so much asking for evidence as… assuming there was none. That tends to annoy people, and I think it makes sense on a productive-discourse level to discourage such posts. On a teach-the-newcomer-and-don’t-drive-them-out-for-not-being-perfectly-rational-yet level, however, I think you have a point.
As an intuition pump, imagine that to become “fully human”, whatever that means, the AI needs to have three traits, let’s call them “A”, “B”, and “C”, whatever those could be. It seems unlikely that the AI would gain all those three traits in the same version. More likely, first we will get an AI 1.0 that has the trait “A”, but lacks traits “B” and “C”. At some later time, we will have an AI 2.0 that has traits “A” and “B”, but still lacks “C”.
Now, the important thing is that if the AI 1.0 had the trait “A” at human level, the scientific progress at “A” probably still continued, so the AI 2.0 most likely already has the trait “A” at super-human level. So it has super-human “A”, human-level “B”, but still lacks “C”.
And for the same reason, when later we have an AI 3.0 that finally has all the traits “A”, “B”, and “C” at least at the human level, it will likely already have traits “A” and “B” at super-human level; the trait “A” probably at insanely-super-human level. In other words, the first “fully human” AI will actually already be super-human in some aspects, simply because it is unlikely that all aspects would reach the human level at the same time.
For example, the first “fully human” AI will easily win the chess tournaments, simply because current AIs can already win them. Instead of some average Joe, the first “fully human” AI will be, at least, some kind of super-Kasparov.
How dangerous it is to have some super-human traits, while being “fully human” otherwise? Depends on the trait. Notice how smarter can people act when you simply give them more time, or better memory (e.g. having a paper, or some personal wiki software), or ability to work in groups. If the AI is otherwise like a human, except e.g. 100 times faster, or able to keep a wiki in its head, or able to do real multitasking (i.e. split into a few independent processes, and explore the issue from multiple angles at the same time), that could already make it quite smart. (Imagine how your life would change if at any moment you could magically take an extra hour to think about stuff, having all relevant books in your head, and an invisible team of equally smart discussion partners.) And these are just quite boring examples of human traits; it could also be e.g. 100 times greater ability to recognize patterns; or perhaps some trait we don’t have a word for yet.
Generally, the idea is that “an average human” is a tiny dot on the intelligence scale. If you make small jumps upwards on the scale, the chance that one of your jumps will end exactly at this dot is very small. More likely, your jumps will repeatedly end below this dot, until at some moment the following jump will take over this dot, at some higher place. Humans have many “hardware” limitations that keep them in a narrow interval, such as brains made of meat working at frequency 200 Hz, or heads small enough to allow childbirth. The AI will have none of these. It will have its own hardware limits, but those will follow different rules. So it seems possible that e.g. the AI built in 2022 will work at human equivalent of 20 Hz, and the AI built in 2023 will work at human equivalent of 2000 Hz, simply because someone invented a smart algorithm allowing much faster simulation of neurons, or used a network of thousand computers instead of one supercomputer, or coded the critical parts in C++ instead of Lisp and had them run on GPU, etc.
But perhaps it is enough to accept that the first “fully human” AI could beat you at chess even in its sleep, and ask yourself what is the chance that chess would the only such example.
“For example, the first “fully human” AI will easily win the chess tournaments, simply because current AIs can already win them.”
No they can’t. *Chess playing programs* can easily win tournaments, self-driving cars and sentiment analysers can’t. An AGI that had the ability to run a chess playing program would be able to win, but the same applies to humans with the same ability.
+1 with this disagreement. ML methods seem to indicate that it’s not quite that simple—they don’t soar beyond human level just from self-improvement, when pointed at themselves they self-improve but relatively very slowly compared to the predictions. I do think it’s possible, but it no longer seems like an obvious thing the way it may have before ML became a real thing.
With respect to being a single system: it’s consistently the case that end-to-end learned neural networks are better at their jobs than plugging together disparately trained networks, which are in turn better at their jobs than neural networks that can only communicate via one-hot vectors (eg, agent populations communicating in english, or ai systems like the heavily multimodal alexa bots). The latter can be much easier to make, but in turn top out much earlier.