I am well aware of Amdahl’s law. By intelligence explosion I mean the period where progress in AI R&D capabilities is at least 10x faster than it is now; it need not be infinitely faster. I agree it’ll be bottlenecked on the slowest component, but according to my best estimates and guesses the overall speed will still be >10x what it is today.
I also agree that there aren’t enough robots; I expect a period of possibly several years between ASI and the surface of the earth being covered in robots.
intelligence explosion I mean the period where progress in AI R&D capabilities is at least 10x faster than it is now; it need not be infinitely faster
Well a common view is that AI now is only possible with the compute levels available recently. This would mean it’s already compute limited.
Say for the sake of engagement that GPTs are inefficient by 10 times vs the above estimate, that “AI progress” is leading to more capabilities on equal compute, and other progress is leading to more capabilities with more compute.
So in this scenario needs 8 H100s per robot. You want to “cover the earth”. 8 billion people don’t cover the earth, let’s assume that 800 billion robots will.
Well how fast will progress happen? Let’s assume we started with a baseline of 2 million H100s in 2024, and each year after we add 50 percent more production rate to the year prior, and every 2.5 years we double the performance per unit, then in 94 years we will have built enough H100s to cover the earth with robots.
How long do you think it requires for a factory to build the components used in itself, including the workers? That would be another way to calculate this. If it’s 2 years, and we started with 250k capable robots (because of H100 shortage) then it would take 43 years for 800 billion. 22 years if it drops to 1 year.
People do bring up biological doubling times but neglect the surrounding ecosystem. When we say “double a robot” we don’t mean theres a table of robot parts and one robot assembles another. That would take a few hours max. It means building all parts, mining for materials, and also doubling all the machinery used to make everything. Double the factories, double the mines, double the power generators.
You can use China’s rate of industrial expansion as a proxy for this. At 15 percent thats a doubling time of 5 years. So if robots double in 2 years they are more than than twice as efficient as China, and remember China got some 1 time bonuses from a large existing population and natural resources like large untapped rivers.
Cellular self replication is using surrounding materials conveniently dissolved in solution, lichen for instance add only a few mm per year.
“AI progress” is then time bound to the production of compute and robots. You make AI progress, once you collect some “1 time” low hanging fruit bonus, at a rate equal to the log(available compute). This is compute you can spare not tied up in robots.
What log base do you assume? What’s the log base for compute vs capabilities measurer at now? Is 1-2 years doubling time reasonable or do you have some data that would support a faster rate of growth? What do you think the “low hanging fruit” bonus is? I assumed 10x above because smaller models seem to currently have to throw away key capabilities to Goodheart benchmarks.
Numbers and the quality of the justification really really matter here. The above says that we shouldn’t be worried about foom, but of course good estimates for the parameters would convince me otherwise.
The above does not say that we shouldn’t be worried about foom. Foom happens inside a lab with a fixed amount of compute; then, the ASIs in the lab direct the creation of more fabs, factories, wetlabs, etc. until the economy is transformed. Indeed the second stage of this process might take several years. I expect it to go at maybe about twice the speed of China initially but to accelerate rather than slow down.
I expect it to go at maybe about twice the speed of China initially
Ok this sounds like a fair estimate. so 30% annual growth rate, or 2.4 years. I estimated 2 years, which would be faster growth. One concrete reason to think that this is a fair estimate is that AI controlled robots will work 24 hours a day. China’s brutal 996 work schedule, which is 72 hours a week, is still only 43% duty cycle—a robot can work at at least a 95% duty cycle. (other 5% is swapping parts as parts of the robot fail)
So trying to engage with the ‘foom’ idea, you develop in the lab some ‘ASI’. That ASI is ‘directing’ these things? Why? Why do humans trust the outputs?
I was thinking the ‘foom’ idea was ok, the ASI robs lies and steals and cons. It finds some way to scam humans into doing work for it. And ok, every 2.4 years it’s resources are doubling. (there might be some early ‘low hanging fruit’, for example people have suggested an ASI might be able to extract money from financial markets with better trades. This will saturate)
Meanwhile, other humans use the same technology that ASI unlocks, and make controlled ASI—models you prompt without context or memory, weaker or distilled ASI models that are specialized. Then they simultaneously, under human direction, invest hundreds of billions of dollars into robot factories and double the equipment available to humans. Say the humans are less efficient, well, how much less efficient? Is there doubling time 3 years? 5?
There is a set of numbers where the ASI win, but so long as the initial starting point for the humans is a large enough resource multiplier, and the humans eventually notice the actions of the rebel ASI, it’s going to usually be human victory. For example:
ASI steals 1 billion dollars and covertly starts doubling. Doubling time is 2 years, 10 years later, they have 32 billion in resources.
Humans do some funding rounds and get 100 billion dollars and legally start doubling. Doubling time is 4 years. 10 years later, they have 400 billion in resources.
Humans spot the unregistered factories belonging to the ASI via IR or serendipity. The first round of AI wars starts...
ASI needs a pretty large utility modifier to win with such a resource disparity, especially as humans can keep feeding the same context for each battle to the ASI models the humans have, stripping off the context bits that let the ASI know when it’s being Ender’s Gamed*, and getting solutions to the tactical scenarios.
I was understanding that ‘foom’ meant somewhat more science fiction takeoff speeds, such as doubling times of a week. That would be an issue if humans do not also have the ability to order their resources doubled on a weekly basis.
I am well aware of Amdahl’s law. By intelligence explosion I mean the period where progress in AI R&D capabilities is at least 10x faster than it is now; it need not be infinitely faster. I agree it’ll be bottlenecked on the slowest component, but according to my best estimates and guesses the overall speed will still be >10x what it is today.
I also agree that there aren’t enough robots; I expect a period of possibly several years between ASI and the surface of the earth being covered in robots.
Well a common view is that AI now is only possible with the compute levels available recently. This would mean it’s already compute limited.
Say for the sake of engagement that GPTs are inefficient by 10 times vs the above estimate, that “AI progress” is leading to more capabilities on equal compute, and other progress is leading to more capabilities with more compute.
So in this scenario needs 8 H100s per robot. You want to “cover the earth”. 8 billion people don’t cover the earth, let’s assume that 800 billion robots will.
Well how fast will progress happen? Let’s assume we started with a baseline of 2 million H100s in 2024, and each year after we add 50 percent more production rate to the year prior, and every 2.5 years we double the performance per unit, then in 94 years we will have built enough H100s to cover the earth with robots.
How long do you think it requires for a factory to build the components used in itself, including the workers? That would be another way to calculate this. If it’s 2 years, and we started with 250k capable robots (because of H100 shortage) then it would take 43 years for 800 billion. 22 years if it drops to 1 year.
People do bring up biological doubling times but neglect the surrounding ecosystem. When we say “double a robot” we don’t mean theres a table of robot parts and one robot assembles another. That would take a few hours max. It means building all parts, mining for materials, and also doubling all the machinery used to make everything. Double the factories, double the mines, double the power generators.
You can use China’s rate of industrial expansion as a proxy for this. At 15 percent thats a doubling time of 5 years. So if robots double in 2 years they are more than than twice as efficient as China, and remember China got some 1 time bonuses from a large existing population and natural resources like large untapped rivers.
Cellular self replication is using surrounding materials conveniently dissolved in solution, lichen for instance add only a few mm per year.
“AI progress” is then time bound to the production of compute and robots. You make AI progress, once you collect some “1 time” low hanging fruit bonus, at a rate equal to the log(available compute). This is compute you can spare not tied up in robots.
What log base do you assume? What’s the log base for compute vs capabilities measurer at now? Is 1-2 years doubling time reasonable or do you have some data that would support a faster rate of growth? What do you think the “low hanging fruit” bonus is? I assumed 10x above because smaller models seem to currently have to throw away key capabilities to Goodheart benchmarks.
Numbers and the quality of the justification really really matter here. The above says that we shouldn’t be worried about foom, but of course good estimates for the parameters would convince me otherwise.
The above does not say that we shouldn’t be worried about foom. Foom happens inside a lab with a fixed amount of compute; then, the ASIs in the lab direct the creation of more fabs, factories, wetlabs, etc. until the economy is transformed. Indeed the second stage of this process might take several years. I expect it to go at maybe about twice the speed of China initially but to accelerate rather than slow down.
Ok this sounds like a fair estimate. so 30% annual growth rate, or 2.4 years. I estimated 2 years, which would be faster growth. One concrete reason to think that this is a fair estimate is that AI controlled robots will work 24 hours a day. China’s brutal 996 work schedule, which is 72 hours a week, is still only 43% duty cycle—a robot can work at at least a 95% duty cycle. (other 5% is swapping parts as parts of the robot fail)
So trying to engage with the ‘foom’ idea, you develop in the lab some ‘ASI’. That ASI is ‘directing’ these things? Why? Why do humans trust the outputs?
I was thinking the ‘foom’ idea was ok, the ASI robs lies and steals and cons. It finds some way to scam humans into doing work for it. And ok, every 2.4 years it’s resources are doubling. (there might be some early ‘low hanging fruit’, for example people have suggested an ASI might be able to extract money from financial markets with better trades. This will saturate)
Meanwhile, other humans use the same technology that ASI unlocks, and make controlled ASI—models you prompt without context or memory, weaker or distilled ASI models that are specialized. Then they simultaneously, under human direction, invest hundreds of billions of dollars into robot factories and double the equipment available to humans. Say the humans are less efficient, well, how much less efficient? Is there doubling time 3 years? 5?
There is a set of numbers where the ASI win, but so long as the initial starting point for the humans is a large enough resource multiplier, and the humans eventually notice the actions of the rebel ASI, it’s going to usually be human victory. For example:
ASI steals 1 billion dollars and covertly starts doubling. Doubling time is 2 years, 10 years later, they have 32 billion in resources.
Humans do some funding rounds and get 100 billion dollars and legally start doubling. Doubling time is 4 years. 10 years later, they have 400 billion in resources.
Humans spot the unregistered factories belonging to the ASI via IR or serendipity. The first round of AI wars starts...
ASI needs a pretty large utility modifier to win with such a resource disparity, especially as humans can keep feeding the same context for each battle to the ASI models the humans have, stripping off the context bits that let the ASI know when it’s being Ender’s Gamed*, and getting solutions to the tactical scenarios.
I was understanding that ‘foom’ meant somewhat more science fiction takeoff speeds, such as doubling times of a week. That would be an issue if humans do not also have the ability to order their resources doubled on a weekly basis.
*c’mon give me credit for this turn of phrase