“Better” in this case depends on the AI’s original utility function. An AI is “better” than its predecessor if it can score more highly on that utility function. As you said, better is arbitrary, but “better with respect to a particular goal” isn’t.
In addition, things like speed, efficiency, correctness, having more copies of yourself, and cognitive capacity are all useful for a wide variety of goals—an AI that is ten times as powerful, ten times faster, significantly more accurate, or capable of understanding more sophisticated concepts as compared to another AI with the same goal is likely to be better at achieving it. This is true for almost any goal you can imagine, whether it be “Create paperclips” or “Maximise human flourishing”.
So, again, how can an AI know which child AI will best fulfill its utility function without being able to observe them in action, which for any sort of long-term goal is necessarily going to be slow?
If the goal is “make the most paperclips per day” the minimum cycle time to evaluate the performance of a child AI is one day. Maybe more if it wants an n > 1 to sample from each AI. And where is this AI getting the resources to try multiple AIs worth of systems? And also, too, if the goal is something complex like “acquire power and/or money” you now have multiple child AIs competing with each-other to better extract resources, which is a different environment than a single AI working to maximize its power/$.
“Better” in this case depends on the AI’s original utility function. An AI is “better” than its predecessor if it can score more highly on that utility function. As you said, better is arbitrary, but “better with respect to a particular goal” isn’t.
In addition, things like speed, efficiency, correctness, having more copies of yourself, and cognitive capacity are all useful for a wide variety of goals—an AI that is ten times as powerful, ten times faster, significantly more accurate, or capable of understanding more sophisticated concepts as compared to another AI with the same goal is likely to be better at achieving it. This is true for almost any goal you can imagine, whether it be “Create paperclips” or “Maximise human flourishing”.
So, again, how can an AI know which child AI will best fulfill its utility function without being able to observe them in action, which for any sort of long-term goal is necessarily going to be slow? If the goal is “make the most paperclips per day” the minimum cycle time to evaluate the performance of a child AI is one day. Maybe more if it wants an n > 1 to sample from each AI. And where is this AI getting the resources to try multiple AIs worth of systems? And also, too, if the goal is something complex like “acquire power and/or money” you now have multiple child AIs competing with each-other to better extract resources, which is a different environment than a single AI working to maximize its power/$.