I think it will be a more intuitive abstraction to talk about an AI that designs the next (at the object level) AI using itself as a template. None of us has conceptual experience with directly modifying our own brains. All of us have built something.
We also have good natural language tools to deal with things like ‘grandchildren’ and ‘inheriting the knowledge base’.
Modifying code in place would be a more likely reality, but I don’t think it’s relevant to the argument, and just makes the whole thing harder to think about.
Yes we’re talking about a self modifying AI design, but each iteration is in many ways a new entity.
This also puts the sexual selection as brains designing brains into perspective. A proto-woman could only choose among the men available to her, which were the result of natural selection “searching the immediate neighborhood of” the previous generation. She can nudge evolution, but she can’t use her insight to make a substantially better mate. ( I think this is ‘metacognitive’ vs. ‘object level’
Even to the degree that there was a feedback loop, it didn’t just have ‘smart’ as its goal. Brains designing brains doesn’t get you foom if they’re designing/selecting for ‘funny’ or ‘apparently faithful but not really’.
I think it will be a more intuitive abstraction to talk about an AI that designs the next (at the object level) AI using itself as a template. None of us has conceptual experience with directly modifying our own brains. All of us have built something. We also have good natural language tools to deal with things like ‘grandchildren’ and ‘inheriting the knowledge base’. Modifying code in place would be a more likely reality, but I don’t think it’s relevant to the argument, and just makes the whole thing harder to think about.
Yes we’re talking about a self modifying AI design, but each iteration is in many ways a new entity.
This also puts the sexual selection as brains designing brains into perspective. A proto-woman could only choose among the men available to her, which were the result of natural selection “searching the immediate neighborhood of” the previous generation. She can nudge evolution, but she can’t use her insight to make a substantially better mate. ( I think this is ‘metacognitive’ vs. ‘object level’
Even to the degree that there was a feedback loop, it didn’t just have ‘smart’ as its goal. Brains designing brains doesn’t get you foom if they’re designing/selecting for ‘funny’ or ‘apparently faithful but not really’.