Waving my hands: For AI to explode, it needs to have a set of capabilities such that for each capability C₁ in the set, there exists some other capability C₂ in the set such that C₂ can be used to improve C₁. (Probably really a set of capabilities would be necessary instead of a single capability C₂, but whatever.)
It may be that the minimal set of capabilities satisfying this requirement is very large. So the AI bootstrapping approach could fail on an AI with a limited set of capabilities, but succeed on an AI with a larger set of capabilities.
I suspect the real problem for a would-be exploding AI is that C₂ is going to be “a new chip foundry worth $20 billion”. Even if the AI can design the plant, and produce enough value that it can buy the plant itself(and that we grant it the legal personhood necessary to do so), it’s not going to happen on a Tuesday evening.
Yeah, I agree that this is a strong possibility, as I wrote in this essay. Parts of it are wrong, but I think it has a few good ideas, especially this bit:
When the AI is as smart as all the world’s AI researchers working together, it will produce new AI insights at the rate that all the world’s AI researchers working together produce new insights. At some point our AGI will be just as smart as the world’s AI researchers, but we can hardly expect to start seeing super-fast AI progress at that point, because the world’s AI researchers haven’t produced super-fast AI progress.
Waving my hands: For AI to explode, it needs to have a set of capabilities such that for each capability C₁ in the set, there exists some other capability C₂ in the set such that C₂ can be used to improve C₁. (Probably really a set of capabilities would be necessary instead of a single capability C₂, but whatever.)
It may be that the minimal set of capabilities satisfying this requirement is very large. So the AI bootstrapping approach could fail on an AI with a limited set of capabilities, but succeed on an AI with a larger set of capabilities.
I suspect the real problem for a would-be exploding AI is that C₂ is going to be “a new chip foundry worth $20 billion”. Even if the AI can design the plant, and produce enough value that it can buy the plant itself(and that we grant it the legal personhood necessary to do so), it’s not going to happen on a Tuesday evening.
Yeah, I agree that this is a strong possibility, as I wrote in this essay. Parts of it are wrong, but I think it has a few good ideas, especially this bit: