Yes. This is known as the question of “takeoff speed”—how fast will an AI of near-human intellect reach superintelligence? “Fast takeoff” suggests that an AI will be able to recursively improve itself—i.e, if an AI is better at coding AI than we are, it can make a better AI, which can make a better AI, and so on, and this could happen across weeks, days, or even hours.
“Slow takeoff” suggests this is not possible or likely, and that we’re likely to take years or decades to move from near-human AGI to smart human AGI to superhuman AGI to superintelligence. (where I use “superintelligence” here to mean “Chimps are to humans are humans are to it”)
I’ve heard people make the argument that AGI takeoff will look like AlphaZero—we spent thousands of years learning to be better at Go, and AlphaZero trained from nothing to superhuman in a matter of days. If this could generalise across the physical world, it would indicate a fast takeoff being more likely.
My question is more pre-takeoff. Specifically, how does an AI that is making additional AIs even know when it has one “better” than itself? What does it even mean to be “better” at making an AI?
And none of that addresses the completely separate question of an AI picking which task to do. A program that recursively writes programs that write programs that are faster/smaller/some other objective metric makes sense. But what good is having the smallest, fastest, small+fast program writer possible?
“Better” in this case depends on the AI’s original utility function. An AI is “better” than its predecessor if it can score more highly on that utility function. As you said, better is arbitrary, but “better with respect to a particular goal” isn’t.
In addition, things like speed, efficiency, correctness, having more copies of yourself, and cognitive capacity are all useful for a wide variety of goals—an AI that is ten times as powerful, ten times faster, significantly more accurate, or capable of understanding more sophisticated concepts as compared to another AI with the same goal is likely to be better at achieving it. This is true for almost any goal you can imagine, whether it be “Create paperclips” or “Maximise human flourishing”.
So, again, how can an AI know which child AI will best fulfill its utility function without being able to observe them in action, which for any sort of long-term goal is necessarily going to be slow?
If the goal is “make the most paperclips per day” the minimum cycle time to evaluate the performance of a child AI is one day. Maybe more if it wants an n > 1 to sample from each AI. And where is this AI getting the resources to try multiple AIs worth of systems? And also, too, if the goal is something complex like “acquire power and/or money” you now have multiple child AIs competing with each-other to better extract resources, which is a different environment than a single AI working to maximize its power/$.
Yes. This is known as the question of “takeoff speed”—how fast will an AI of near-human intellect reach superintelligence? “Fast takeoff” suggests that an AI will be able to recursively improve itself—i.e, if an AI is better at coding AI than we are, it can make a better AI, which can make a better AI, and so on, and this could happen across weeks, days, or even hours.
“Slow takeoff” suggests this is not possible or likely, and that we’re likely to take years or decades to move from near-human AGI to smart human AGI to superhuman AGI to superintelligence. (where I use “superintelligence” here to mean “Chimps are to humans are humans are to it”)
I’ve heard people make the argument that AGI takeoff will look like AlphaZero—we spent thousands of years learning to be better at Go, and AlphaZero trained from nothing to superhuman in a matter of days. If this could generalise across the physical world, it would indicate a fast takeoff being more likely.
My question is more pre-takeoff. Specifically, how does an AI that is making additional AIs even know when it has one “better” than itself? What does it even mean to be “better” at making an AI? And none of that addresses the completely separate question of an AI picking which task to do. A program that recursively writes programs that write programs that are faster/smaller/some other objective metric makes sense. But what good is having the smallest, fastest, small+fast program writer possible?
“Better” in this case depends on the AI’s original utility function. An AI is “better” than its predecessor if it can score more highly on that utility function. As you said, better is arbitrary, but “better with respect to a particular goal” isn’t.
In addition, things like speed, efficiency, correctness, having more copies of yourself, and cognitive capacity are all useful for a wide variety of goals—an AI that is ten times as powerful, ten times faster, significantly more accurate, or capable of understanding more sophisticated concepts as compared to another AI with the same goal is likely to be better at achieving it. This is true for almost any goal you can imagine, whether it be “Create paperclips” or “Maximise human flourishing”.
So, again, how can an AI know which child AI will best fulfill its utility function without being able to observe them in action, which for any sort of long-term goal is necessarily going to be slow? If the goal is “make the most paperclips per day” the minimum cycle time to evaluate the performance of a child AI is one day. Maybe more if it wants an n > 1 to sample from each AI. And where is this AI getting the resources to try multiple AIs worth of systems? And also, too, if the goal is something complex like “acquire power and/or money” you now have multiple child AIs competing with each-other to better extract resources, which is a different environment than a single AI working to maximize its power/$.