I think it’s important to note that a single “human level” AI system can have access to human-level competency in every human cognitive skill and domain simultaneously, and can have read/heard/seen everything humanity has ever output, which is something no biological human has. Whatever limit we approach as the limit of feedback loops, a true omnimath is a very loose lower bound on what human-level AI can achieve even with no self-improvement and no speed advantage. Consider in the context of human polymaths who manage to revolutionize many fields in the course of a single lifetime.
In practice I think we should expect a considerable speed advantage and high number of instances, if not immediately then very soon after whoever develops the system demonstrates its value. Plus, perfect recall and never losing attention/focus and extremely high input/output data compared to human senses seem trivial to add if they’re not there by default.
In other words, individually-human-level thinking skills and levels of knowledge imply significantly superhuman capabilities in an AGI before we even ask the question of how high the limit of fast-self-improvement will be. And then, of course, there’s the question of how superhuman something needs to actually be to present significant x-risk. My take is that this does not require much self-improvement at all, and that a very fast-thinking and internally-coordinated collective of polymathic agents with many individually-human-level capabilties is sufficiently x-risky to be worried about.
The reason I think this is that I believe there are an enormous number of ways to make better use of the research data and experimentally demonstrated capabilities humans already have, if we could instantly combine the relevant skills from different fields in a single individual as needed.
For example, you mention eking out a few more years of Moore’s law by advancing EUV lithography. But, I would be extremely surprised if a human with all the PhDs and a multi-GHz brain couldn’t work out how to produce optical or ultrasonic computing metasurfaces, on already existing not-even-very-new fab equipment, based on variants and combinations of already proposed designs for things like diffractive neural nets, which would operate several orders of magnitude faster than the highest speed electronic chips. In other words, I see “take what humans already know and have the equipment to build today, and use it to speed themselves up by 1000x-100,000x” as a lower bound on self-improvement without any increase in quality of thought compared to humans.
All of these things are possible, but it’s not clear to me that they’re likely, at least in the early stages of AGI. In other words: once we have significantly-superhuman AGI, then agreed, all sorts of crazy things may become possible. But first we have to somehow achieve superhuman AGI. One of the things I’m trying to do in this post is explore the path that gets us to superhuman AGI in the first place. That path, by definition, can’t rely on anything that requires superhuman capabilities.
If I understand correctly, you’re envisioning that we will be able to construct AGIs that have human-level capability, and far greater than human speed, in order to bootstrap superhuman AGI? What makes you confident that this speed advantage will exist early on? Current leading-edge models like GPT-4 are not drastically faster than human beings, and presumably until we get to human-level AGI we’ll be spending most of our algorithmic improvements and increases in FLOP budget on increased capabilities, rather than performance. In fact, it’s quite possible that we’ll have to (temporarily) accept reduced speed in order to achieve human-level performance; for instance, by introducing tree search into the thought process (tree-of-thought prompting, heuristic search techniques in Yann LeCun’s “A Path Towards Autonomous Machine Intelligence”, etc.).
Once we achieve human-level, human-speed AGI, then yes, further algorithm or FLOPs improvements could be spent on speed; this comes back to the basic question of how whether the cognitive effort required for further progress increases more or less rapidly than the extent to which progress (and/or increased budgets) enables increased cognitive effort, i.e. does the self-improvement feedback loop converge or diverge. Are you proposing that it definitely diverges? What points you in that direction?
I would also caution against being highly confident that AGI will automatically be some sort of ideal omnimath. Such ability would require more than merely assimilating all of human knowledge and abilities; it would require knowing exactly which sub-specialties to draw on in any given moment. Otherwise, the AI would risk drowning in millions of superfluous connections to its every thought. Some examples of human genius might in part depend on a particular individual just happening to have precisely the right combination of knowledge, without a lot of other superfluous considerations to distract them.
Also, is it obvious that a single early-human-level AI could be trained with deep mastery of every field of human knowledge? Biological-anchor analysis aims to project when we can create a human-level AI, and humans are not omnimaths. Deep expertise across every subspeciality might easily require many more parameters than the number of synapses in the human brain. Many things look simple until you try to implement them; I touch on this in a recent blog post, The AI Progress Paradox, but you just need to look at the history of self-driving cars (or photorealistic CGI, or many other examples) to see how things that seem simple in principle can require many rounds of iteration to fully achieve in practice.
If I understand correctly, you’re envisioning that we will be able to construct AGIs that have human-level capability, and far greater than human speed, in order to bootstrap superhuman AGI? What makes you confident that this speed advantage will exist early on?
What I’m trying to say is that even at human speed, being able to mix-and-match human-level capabilities at will, in arbitrary combinations, not an ideal omnimath but in larger numbers than a single human can accumulate, is already a superhuman ability and one I expect AGI to trivially possess. Then on top of that you get, for free, things like being able to coordinate multiple instances of a single entity that don’t have their own other agendas, and that never lose focus or get tired.
Since you did mention genius coming from “precisely the right combination of knowledge, without a lot of other superfluous considerations to distract them,” I have to ask… doesn’t AGI seem perfectly positioned to be just that, for any combination of knowledge you can train it on?
I also don’t find the biological anchors argument convincing, for somewhat the same reason: an AI doesn’t need all of the superfluous knowledge a human has. Some of it, yes, but not all of it. To put it another way, in terms of data and parameters, how much knowledge of physics does a physicist actually have after a long career? A basic world model like all humans acquire in childhood, plus a few hundred books, a few thousand hours of lectures, and maybe 40k hours of sensory data acquired and thinking completed on-the-job?
And you’re right, I agree an early AGI won’t be an omnimath, but I think polymath is very much within reach.
I think it’s important to note that a single “human level” AI system can have access to human-level competency in every human cognitive skill and domain simultaneously, and can have read/heard/seen everything humanity has ever output, which is something no biological human has. Whatever limit we approach as the limit of feedback loops, a true omnimath is a very loose lower bound on what human-level AI can achieve even with no self-improvement and no speed advantage. Consider in the context of human polymaths who manage to revolutionize many fields in the course of a single lifetime.
In practice I think we should expect a considerable speed advantage and high number of instances, if not immediately then very soon after whoever develops the system demonstrates its value. Plus, perfect recall and never losing attention/focus and extremely high input/output data compared to human senses seem trivial to add if they’re not there by default.
In other words, individually-human-level thinking skills and levels of knowledge imply significantly superhuman capabilities in an AGI before we even ask the question of how high the limit of fast-self-improvement will be. And then, of course, there’s the question of how superhuman something needs to actually be to present significant x-risk. My take is that this does not require much self-improvement at all, and that a very fast-thinking and internally-coordinated collective of polymathic agents with many individually-human-level capabilties is sufficiently x-risky to be worried about.
The reason I think this is that I believe there are an enormous number of ways to make better use of the research data and experimentally demonstrated capabilities humans already have, if we could instantly combine the relevant skills from different fields in a single individual as needed.
For example, you mention eking out a few more years of Moore’s law by advancing EUV lithography. But, I would be extremely surprised if a human with all the PhDs and a multi-GHz brain couldn’t work out how to produce optical or ultrasonic computing metasurfaces, on already existing not-even-very-new fab equipment, based on variants and combinations of already proposed designs for things like diffractive neural nets, which would operate several orders of magnitude faster than the highest speed electronic chips. In other words, I see “take what humans already know and have the equipment to build today, and use it to speed themselves up by 1000x-100,000x” as a lower bound on self-improvement without any increase in quality of thought compared to humans.
All of these things are possible, but it’s not clear to me that they’re likely, at least in the early stages of AGI. In other words: once we have significantly-superhuman AGI, then agreed, all sorts of crazy things may become possible. But first we have to somehow achieve superhuman AGI. One of the things I’m trying to do in this post is explore the path that gets us to superhuman AGI in the first place. That path, by definition, can’t rely on anything that requires superhuman capabilities.
If I understand correctly, you’re envisioning that we will be able to construct AGIs that have human-level capability, and far greater than human speed, in order to bootstrap superhuman AGI? What makes you confident that this speed advantage will exist early on? Current leading-edge models like GPT-4 are not drastically faster than human beings, and presumably until we get to human-level AGI we’ll be spending most of our algorithmic improvements and increases in FLOP budget on increased capabilities, rather than performance. In fact, it’s quite possible that we’ll have to (temporarily) accept reduced speed in order to achieve human-level performance; for instance, by introducing tree search into the thought process (tree-of-thought prompting, heuristic search techniques in Yann LeCun’s “A Path Towards Autonomous Machine Intelligence”, etc.).
Once we achieve human-level, human-speed AGI, then yes, further algorithm or FLOPs improvements could be spent on speed; this comes back to the basic question of how whether the cognitive effort required for further progress increases more or less rapidly than the extent to which progress (and/or increased budgets) enables increased cognitive effort, i.e. does the self-improvement feedback loop converge or diverge. Are you proposing that it definitely diverges? What points you in that direction?
I would also caution against being highly confident that AGI will automatically be some sort of ideal omnimath. Such ability would require more than merely assimilating all of human knowledge and abilities; it would require knowing exactly which sub-specialties to draw on in any given moment. Otherwise, the AI would risk drowning in millions of superfluous connections to its every thought. Some examples of human genius might in part depend on a particular individual just happening to have precisely the right combination of knowledge, without a lot of other superfluous considerations to distract them.
Also, is it obvious that a single early-human-level AI could be trained with deep mastery of every field of human knowledge? Biological-anchor analysis aims to project when we can create a human-level AI, and humans are not omnimaths. Deep expertise across every subspeciality might easily require many more parameters than the number of synapses in the human brain. Many things look simple until you try to implement them; I touch on this in a recent blog post, The AI Progress Paradox, but you just need to look at the history of self-driving cars (or photorealistic CGI, or many other examples) to see how things that seem simple in principle can require many rounds of iteration to fully achieve in practice.
What I’m trying to say is that even at human speed, being able to mix-and-match human-level capabilities at will, in arbitrary combinations, not an ideal omnimath but in larger numbers than a single human can accumulate, is already a superhuman ability and one I expect AGI to trivially possess. Then on top of that you get, for free, things like being able to coordinate multiple instances of a single entity that don’t have their own other agendas, and that never lose focus or get tired.
Since you did mention genius coming from “precisely the right combination of knowledge, without a lot of other superfluous considerations to distract them,” I have to ask… doesn’t AGI seem perfectly positioned to be just that, for any combination of knowledge you can train it on?
I also don’t find the biological anchors argument convincing, for somewhat the same reason: an AI doesn’t need all of the superfluous knowledge a human has. Some of it, yes, but not all of it. To put it another way, in terms of data and parameters, how much knowledge of physics does a physicist actually have after a long career? A basic world model like all humans acquire in childhood, plus a few hundred books, a few thousand hours of lectures, and maybe 40k hours of sensory data acquired and thinking completed on-the-job?
And you’re right, I agree an early AGI won’t be an omnimath, but I think polymath is very much within reach.