I am genuinely curious and confused as to what exactly you concretely imagine this supposed ‘superintelligence’ to be, such that is not already the size of a factory, such that you mention “size of a factory” as if that is something actually worth mentioning—at all. Please show at least your first pass fermi estimates for the compute requirements.
Despite your claim to be “genuinely curious and confused,” the overarching tone and content of this bit does not strike me as projecting curiosity or confusion, but instead confident and sharp-toned burden-of-proof-shifting to habryka. That’s merely a stylistic note, not impacting the content of your claims.
It sounds here like you are agreeing with him that you can deal with any limits to ops/mm^3 limits by simply building a bigger computer. It’s therefore hard for me to see why these arguments about efficiency limitations matter very much for AI’s ability to be superintelligent and exhibit superhuman takeover capabilities.
I can see why maybe human brains, being efficient according to certain metrics, might be a useful tool for the AI to keep around, but I don’t see why we ought to feel at all reassured by that. I don’t really want to serve out the end of my days as an AI’s robot.
I would suggest reading more about advanced GPU/accelerator design, and then about datacenter design and the thermodynamic/cooling considerations therein.
Just as a piece of feedback, this sort of comment is not very enlightening, and it doesn’t convince me that you have background knowledge that ought to make me believe you and not habryka. If your aim is to “teach the crowd,” N of 1, but I am not being effectively taught by this bit.
A quick google search indicates 1GW is a typical power plant output, which in theory could power roughly a million GPU datacenter. This is almost 100 times larger in power consumption than the current largest official supercomputer: Frontier—which has about 30k GPUs. The supercomputer used to train GPT4 is somewhat of a secret, but estimated to be about that size. So at 50x to 100x you are talking about scaling up to something approaching a hypothetical GPT-5 scale cluster.
So it takes 1% of a single power plant output to train GPT4. If GPT-4 got put on chips and distributed, wouldn’t it take only a very small amount of power comparatively to actually run it once trained? Why are we talking about training costs rather than primarily about the cost to operate models once they have been trained?
But no, I do not think that this hypothetical not currently achievable GPT5 - even if you were running 100k instances of it—would “likely be easily capable of disempowering humanity”.′
You have output a lot of writings on this subject and I’ve only read a fraction. Do you argue this point somewhere? This seems pretty cruxy. In my model, at this point, doomers are partly worried about what happens during training of larger and larger models, but they’re also worried about what happens when you proliferate many copies of extant models and put them into something like an AutoGPT format, where they can make plans, access real-world resources, and use plugins.
Again, I think trying to focus the conversation on efficiency by itself is not enough. Right now, I feel like you have a very deep understanding of efficiency, but that once the conversation moves away from the topic of efficiency along some metric and into topics like “why can’t AI just throw more resources at expanding itself” or “why can’t AI take over with a bunch of instances of ChatGPT 4 or 5 in an AutoGPT context” or “why should I be happy about ending my days as the AI’s efficient meat-brain robot,” it becomes hard for me to follow your argument or understand why it’s an important “contra Yudkowsky” or “contra central Yudkowsky-appreciating doomers” argument.
It feels more like constructing, I don’t know, a valid critique of some of the weakest parts (maybe?) of Yudkowsky’s entire written ouvre having to do with the topic of efficiency and nanobots, and then saying that because this fails, the entire Yudkowskian argument about AI doom fails, or even that arguments for AI takeover scenarios fail generally. And I am not at all convinced you’ve shown that it’s on the foundation of Yudkowsky’s (mis?)understanding of efficiency that the rest of his argument stands or falls.
I’ve never feared grey goo or diamondoid bacteria. Mainly, I worry that a malevolent AI might not even need very much intelligence to steamroll humanity. It may just need a combination of the willingness to do damage, the ability to be smarter than a moderately intelligent person in order to hack our social systems, and the ability to proliferate across our computing infrastructure while not caring about what happens to “itself” should individual instances get “caught” and deleted. So much of our ability to mitigate terrorism depends on the ability to catch and punish offenders, the idea that perpetrators care about their own skin and are confined within it. It is the alienness of the form of the AI, and its ability to perform on par with or exceed performance of humans in important areas, to expand the computing and material resources available to itself, that make it seem dangerous to me.
Despite your claim to be “genuinely curious and confused,” the overarching tone and content of this bit does not strike me as projecting curiosity or confusion, but instead confident and sharp-toned burden-of-proof-shifting to habryka
I see how that tone could come off as rude, but really I don’t understand habryka’s model when he says “a superintelligence will just make a brain the size of a factory, and then be in a position to outcompete or destroy humanity quite easily.”
So it takes 1% of a single power plant output to train GPT4. If GPT-4 got put on chips and distributed, wouldn’t it take only a very small amount of power comparatively to actually run it once trained? Why are we talking about training costs rather than primarily about the cost to operate models once they have been trained?
The transformer arch is fully parallelizable only during training, but is roughly just as or more inefficient than RNNs on GPUs/accelerators for inference. The inference costs of GPT4 are of course a openai/microsoft secret, but it is not a cheap model. Also human-level AGI, let alone superintelligence, will likely require continual learning/training.
I guess by “put on chips” you mean baking GPT-4 into an ASIC? That usually doesn’t make sense for fast changing tech, but as moore’s law slows we could start seeing that. I would expect other various changes to tensorcores well before that.
It feels more like constructing, I don’t know, a valid critique of some of the weakest parts (maybe?) of Yudkowsky’s entire written ouvre having to do with the topic of efficiency and nanobots, and then saying that because this fails, the entire Yudkowskian argument about AI doom fails, or even that arguments for AI takeover scenarios fail generally.
The entire specific Yudkowskian argument about hard takeoff via recursive self improving AGI fooming to nanotech is specifically what i’m arguing against, and indeed I only need to argue against the weakest links.
It is the alienness of the form of the AI
See the section on That Alien Mindspace. DL-based AGI is anthropomorphic (as I predicted), not alien.
Re Yudkowsky, I don’t think his entire argument rests on efficiency, and the pieces that don’t can’t be dispatched by arguing about efficiency.
Regarding “alien mindspace,” what I mean is that the physical form of AI, and whatever awareness the AI has of that, makes it alien. Like, if I knew I could potentially transmit my consciousness with perfect precision over the internet and create self-clones almost effortlessly, I would think very differently than I do now.
His argument entirely depends on efficiency. He claims that near future AGI somewhat smarter than us creates even smarter AGI and so on, recursively bottoming out in something that is many many OOM more intelligent than us without using unrealistic amounts of energy, and all of this happens very quickly.
So that’s entirely an argument that boils down to practical computational engineering efficiency considerations. Additionally he needs the AGI to be unaligned by default, and that argument is also faulty.
EY’s model requires slightly-smarter-than-us AGI running on normal hardware to start a FOOM cycle of recursive self improvement resulting in many OOM intelligence improvement in a short amount of time. That requires some combination of 1.) many OOM software improvement on current hardware, 2.) many OOM hardware improvement with current foundry tech, or 3.) completely new foundry tech with many OOM improvement over current—ie nanotech woo. The viability of all/any of this is all entirely dependent on near term engineering practicality.
It seems like in one place, you’re saying EY’s model depends on near term engineering practicality, and in another, that it depends on physics-constrainted efficiency which you argue invalidates it. Being no expert on the physics-based efficiency arguments, I’m happy to concede the physics constraints. But I’m struggling to understand their relevance to non-physics-based efficiency arguments or their strong bearing on matters of engineering practicality.
My understanding is that your argument goes something like this:
You can’t build something many OOMs more intelligent than a brain on hardware with roughly the same size and energy consumption as the brain.
Therefore, building a superintelligent AI would require investing more energy and more material resources than a brain uses.
Therefore… and here’s where the argument loses steam for me. Why can’t we or the AI just invest lots of material and energy resources? How much smarter than us does an unaligned AI need to be to pose a threat, and why should we think resources are a major constraint to get it to recursively self-improve itself to get to that point? Why should we think it will need constant retraining to recursively self-improve? Why do we think it’ll want to keep an economy going?
As far as the “anthropomorphic” counterargument to the “vast space of alien minds” thing, I fully agree that it appears the easiest way to predict tokens from human text is to simulate a human mind. That doesn’t mean the AI is a human mind, or that it is intrinsically constrained to human values. Being able to articulate those values and imitate behaviors that accord with those values is a capability, not a constraint. We have evidence from things like ChaosGPT or jailbreaks that you can easily have the AI behave in ways that appear unaligned, and that even the appearance of consistent alignment has to be consistently enforced in ways that look awfully fragile.
Overall, my sense is that you’ve admirably spent a lot of time probing the physical limits of certain efficiency metrics and how they bear on AI, and I think you have some intriguing arguments about nanotech and “mindspace” and practical engineering as well.
However, I think your arguments would be more impactful if you carefully and consistently delineated these different arguments and attached them more precisely to the EY claims you’re rebutting, and did more work to show how EY’s conclusion X flows from EY’s argument A, and that A is wrong for efficiency reason B, which overturns X but not Y; you disagree with Y for reason C, overturning EY’s argument D. Right now, I think you do make many of these argumentative moves, but they’re sort of scattered across various posts and comments, and I’m open to the idea that they’re all there but I’ve also seen enough inconsistencies to worry that they’re not. To be clear, I would absolutely LOVE it if EY did the very same thing—the burden of proof should ideally not be all on you, and I maintain uncertainty about this whole issue because of the fragmented nature of the debate.
So at this point, it’s hard for me to update more than “some arguments about efficiency and mindspace and practical engineering and nanotech are big points of contention between Jacob and Eliezer.” I’d like to go further and, with you reject arguments that you believe to be false, but I’m not able to do that yet because of the issue that I’m describing here. While I’m hesitant to burden you with additional work, I don’t have the background or the familiarity with your previous writings to do this very effectively—at the end of the day, if anybody’s going to bring together your argument all in one place and make it crystal clear, I think that person has to be you.
His argument entirely depends on efficiency. He claims that near future AGI somewhat smarter than us creates even smarter AGI and so on, recursively bottoming out in something that is many many OOM more intelligent than us without using unrealistic amounts of energy, and all of this happens very quickly.
You just said in your comment to me that a single power plant is enough to run 100M brains. It seems like you need zero hardware progress in order to get something much smarter without unrealistic amounts of energy, so I just don’t understand the relevance of this.
I said longer term—using hypothetical brain-parity neuromorphic computing (uploads or neuromorphic AGI). We need enormous hardware progress to reach that.
Current tech on GPUs requires large supercomputers to train 1e25+ flops models like GPT4 that are approaching, but not quite, human level AGI. If the rurmour of 1T params is true, then it takes a small cluster and ~10KW just to run some smallish number of instances of the model.
Getting something much much smarter than us would require enormous amounts of computation and energy without large advances in software and hardware.
I said longer term—using hypothetical brain-parity neuromorphic computing (uploads or neuromorphic AGI). We need enormous hardware progress to reach that.
Sure. We will probably get enormous hardware progress over the next few decades, so that’s not really an obstacle.
It seems to me your argument is “smarter than human intelligence cannot make enormous hardware or software progress in a relatively short amount of time”, but this has nothing to do with “efficiency arguments”. The bottleneck is not energy, the bottleneck is algorithmic improvements and improvements to GPU production, neither of which is remotely bottlenecked on energy consumption.
Getting something much much smarter than us would require enormous amounts [...] energy without large advances in software and hardware.
No, as you said, it would require like, a power plant worth of energy. Maybe even like 10 power plants or so if you are really stretching it, but as you said, the really central bottleneck here is GPU production, not energy in any relevant way.
Sure. We will probably get enormous hardware progress over the next few decades, so that’s not really an obstacle.
As we get more hardware and slow mostly-aligned AGI/AI progress this further raises the bar for foom.
It seems to me your argument is “smarter than human intelligence cannot make enormous hardware or software progress in a relatively short amount of time”, but this has nothing to do with “efficiency arguments”.
That is actually an efficiency argument, and in my brain efficiency post I discuss multiple sub components of net efficiency that translate into intelligence/$.
The bottleneck is not energy, the bottleneck is algorithmic improvements and improvements to GPU production, neither of which is remotely bottlenecked on energy consumption.
Ahh I see—energy efficiency is tightly coupled to other circuit efficiency metrics as they are all primarily driven by shrinkage. As you increasingly bottom out hardware improvements energy then becomes an increasingly more direct constraint. This is already happening with GPUs where power consumption is roughly doubling with each generation, and could soon dominate operating costs.
See here where I line the roodman model up to future energy usage predictions.
All that being said I do agree that yes the primary bottlneck or crux for the EY fast takeoff/takeover seems to be the amount of slack in software and scaling laws. But only after we agree that there isn’t obvious easy routes for the AGI to bootstrap nanotech assemblers with many OOM greater compute per J than brains or current computers.
Despite your claim to be “genuinely curious and confused,” the overarching tone and content of this bit does not strike me as projecting curiosity or confusion, but instead confident and sharp-toned burden-of-proof-shifting to habryka. That’s merely a stylistic note, not impacting the content of your claims.
It sounds here like you are agreeing with him that you can deal with any limits to ops/mm^3 limits by simply building a bigger computer. It’s therefore hard for me to see why these arguments about efficiency limitations matter very much for AI’s ability to be superintelligent and exhibit superhuman takeover capabilities.
I can see why maybe human brains, being efficient according to certain metrics, might be a useful tool for the AI to keep around, but I don’t see why we ought to feel at all reassured by that. I don’t really want to serve out the end of my days as an AI’s robot.
Just as a piece of feedback, this sort of comment is not very enlightening, and it doesn’t convince me that you have background knowledge that ought to make me believe you and not habryka. If your aim is to “teach the crowd,” N of 1, but I am not being effectively taught by this bit.
So it takes 1% of a single power plant output to train GPT4. If GPT-4 got put on chips and distributed, wouldn’t it take only a very small amount of power comparatively to actually run it once trained? Why are we talking about training costs rather than primarily about the cost to operate models once they have been trained?
You have output a lot of writings on this subject and I’ve only read a fraction. Do you argue this point somewhere? This seems pretty cruxy. In my model, at this point, doomers are partly worried about what happens during training of larger and larger models, but they’re also worried about what happens when you proliferate many copies of extant models and put them into something like an AutoGPT format, where they can make plans, access real-world resources, and use plugins.
Again, I think trying to focus the conversation on efficiency by itself is not enough. Right now, I feel like you have a very deep understanding of efficiency, but that once the conversation moves away from the topic of efficiency along some metric and into topics like “why can’t AI just throw more resources at expanding itself” or “why can’t AI take over with a bunch of instances of ChatGPT 4 or 5 in an AutoGPT context” or “why should I be happy about ending my days as the AI’s efficient meat-brain robot,” it becomes hard for me to follow your argument or understand why it’s an important “contra Yudkowsky” or “contra central Yudkowsky-appreciating doomers” argument.
It feels more like constructing, I don’t know, a valid critique of some of the weakest parts (maybe?) of Yudkowsky’s entire written ouvre having to do with the topic of efficiency and nanobots, and then saying that because this fails, the entire Yudkowskian argument about AI doom fails, or even that arguments for AI takeover scenarios fail generally. And I am not at all convinced you’ve shown that it’s on the foundation of Yudkowsky’s (mis?)understanding of efficiency that the rest of his argument stands or falls.
I’ve never feared grey goo or diamondoid bacteria. Mainly, I worry that a malevolent AI might not even need very much intelligence to steamroll humanity. It may just need a combination of the willingness to do damage, the ability to be smarter than a moderately intelligent person in order to hack our social systems, and the ability to proliferate across our computing infrastructure while not caring about what happens to “itself” should individual instances get “caught” and deleted. So much of our ability to mitigate terrorism depends on the ability to catch and punish offenders, the idea that perpetrators care about their own skin and are confined within it. It is the alienness of the form of the AI, and its ability to perform on par with or exceed performance of humans in important areas, to expand the computing and material resources available to itself, that make it seem dangerous to me.
I see how that tone could come off as rude, but really I don’t understand habryka’s model when he says “a superintelligence will just make a brain the size of a factory, and then be in a position to outcompete or destroy humanity quite easily.”
The transformer arch is fully parallelizable only during training, but is roughly just as or more inefficient than RNNs on GPUs/accelerators for inference. The inference costs of GPT4 are of course a openai/microsoft secret, but it is not a cheap model. Also human-level AGI, let alone superintelligence, will likely require continual learning/training.
I guess by “put on chips” you mean baking GPT-4 into an ASIC? That usually doesn’t make sense for fast changing tech, but as moore’s law slows we could start seeing that. I would expect other various changes to tensorcores well before that.
The entire specific Yudkowskian argument about hard takeoff via recursive self improving AGI fooming to nanotech is specifically what i’m arguing against, and indeed I only need to argue against the weakest links.
See the section on That Alien Mindspace. DL-based AGI is anthropomorphic (as I predicted), not alien.
Re Yudkowsky, I don’t think his entire argument rests on efficiency, and the pieces that don’t can’t be dispatched by arguing about efficiency.
Regarding “alien mindspace,” what I mean is that the physical form of AI, and whatever awareness the AI has of that, makes it alien. Like, if I knew I could potentially transmit my consciousness with perfect precision over the internet and create self-clones almost effortlessly, I would think very differently than I do now.
His argument entirely depends on efficiency. He claims that near future AGI somewhat smarter than us creates even smarter AGI and so on, recursively bottoming out in something that is many many OOM more intelligent than us without using unrealistic amounts of energy, and all of this happens very quickly.
So that’s entirely an argument that boils down to practical computational engineering efficiency considerations. Additionally he needs the AGI to be unaligned by default, and that argument is also faulty.
In your other recent comment to me, you said:
It seems like in one place, you’re saying EY’s model depends on near term engineering practicality, and in another, that it depends on physics-constrainted efficiency which you argue invalidates it. Being no expert on the physics-based efficiency arguments, I’m happy to concede the physics constraints. But I’m struggling to understand their relevance to non-physics-based efficiency arguments or their strong bearing on matters of engineering practicality.
My understanding is that your argument goes something like this:
You can’t build something many OOMs more intelligent than a brain on hardware with roughly the same size and energy consumption as the brain.
Therefore, building a superintelligent AI would require investing more energy and more material resources than a brain uses.
Therefore… and here’s where the argument loses steam for me. Why can’t we or the AI just invest lots of material and energy resources? How much smarter than us does an unaligned AI need to be to pose a threat, and why should we think resources are a major constraint to get it to recursively self-improve itself to get to that point? Why should we think it will need constant retraining to recursively self-improve? Why do we think it’ll want to keep an economy going?
As far as the “anthropomorphic” counterargument to the “vast space of alien minds” thing, I fully agree that it appears the easiest way to predict tokens from human text is to simulate a human mind. That doesn’t mean the AI is a human mind, or that it is intrinsically constrained to human values. Being able to articulate those values and imitate behaviors that accord with those values is a capability, not a constraint. We have evidence from things like ChaosGPT or jailbreaks that you can easily have the AI behave in ways that appear unaligned, and that even the appearance of consistent alignment has to be consistently enforced in ways that look awfully fragile.
Overall, my sense is that you’ve admirably spent a lot of time probing the physical limits of certain efficiency metrics and how they bear on AI, and I think you have some intriguing arguments about nanotech and “mindspace” and practical engineering as well.
However, I think your arguments would be more impactful if you carefully and consistently delineated these different arguments and attached them more precisely to the EY claims you’re rebutting, and did more work to show how EY’s conclusion X flows from EY’s argument A, and that A is wrong for efficiency reason B, which overturns X but not Y; you disagree with Y for reason C, overturning EY’s argument D. Right now, I think you do make many of these argumentative moves, but they’re sort of scattered across various posts and comments, and I’m open to the idea that they’re all there but I’ve also seen enough inconsistencies to worry that they’re not. To be clear, I would absolutely LOVE it if EY did the very same thing—the burden of proof should ideally not be all on you, and I maintain uncertainty about this whole issue because of the fragmented nature of the debate.
So at this point, it’s hard for me to update more than “some arguments about efficiency and mindspace and practical engineering and nanotech are big points of contention between Jacob and Eliezer.” I’d like to go further and, with you reject arguments that you believe to be false, but I’m not able to do that yet because of the issue that I’m describing here. While I’m hesitant to burden you with additional work, I don’t have the background or the familiarity with your previous writings to do this very effectively—at the end of the day, if anybody’s going to bring together your argument all in one place and make it crystal clear, I think that person has to be you.
You just said in your comment to me that a single power plant is enough to run 100M brains. It seems like you need zero hardware progress in order to get something much smarter without unrealistic amounts of energy, so I just don’t understand the relevance of this.
I said longer term—using hypothetical brain-parity neuromorphic computing (uploads or neuromorphic AGI). We need enormous hardware progress to reach that.
Current tech on GPUs requires large supercomputers to train 1e25+ flops models like GPT4 that are approaching, but not quite, human level AGI. If the rurmour of 1T params is true, then it takes a small cluster and ~10KW just to run some smallish number of instances of the model.
Getting something much much smarter than us would require enormous amounts of computation and energy without large advances in software and hardware.
Sure. We will probably get enormous hardware progress over the next few decades, so that’s not really an obstacle.
It seems to me your argument is “smarter than human intelligence cannot make enormous hardware or software progress in a relatively short amount of time”, but this has nothing to do with “efficiency arguments”. The bottleneck is not energy, the bottleneck is algorithmic improvements and improvements to GPU production, neither of which is remotely bottlenecked on energy consumption.
No, as you said, it would require like, a power plant worth of energy. Maybe even like 10 power plants or so if you are really stretching it, but as you said, the really central bottleneck here is GPU production, not energy in any relevant way.
As we get more hardware and slow mostly-aligned AGI/AI progress this further raises the bar for foom.
That is actually an efficiency argument, and in my brain efficiency post I discuss multiple sub components of net efficiency that translate into intelligence/$.
Ahh I see—energy efficiency is tightly coupled to other circuit efficiency metrics as they are all primarily driven by shrinkage. As you increasingly bottom out hardware improvements energy then becomes an increasingly more direct constraint. This is already happening with GPUs where power consumption is roughly doubling with each generation, and could soon dominate operating costs.
See here where I line the roodman model up to future energy usage predictions.
All that being said I do agree that yes the primary bottlneck or crux for the EY fast takeoff/takeover seems to be the amount of slack in software and scaling laws. But only after we agree that there isn’t obvious easy routes for the AGI to bootstrap nanotech assemblers with many OOM greater compute per J than brains or current computers.
How much room is there in algorithmic improvements?
Maybe it would be a good idea to change the title of this essay to:
as to not give people hope that there would be a counter argument somewhere in this article to his more general claim: