I feel like even under the worldview that your beliefs imply, a superintelligence will just make a brain the size of a factory, and then be in a position to outcompete or destroy humanity quite easily.
I am genuinely curious and confused as to what exactly you concretely imagine this supposed ‘superintelligence’ to be, such that is not already the size of a factory, such that you mention “size of a factory” as if that is something actually worth mentioning—at all. Please show at least your first pass fermi estimates for the compute requirements. By that I mean—what are the compute requirements for the initial SI—and then the later presumably more powerful ‘factory’?
Maybe it will do that using GPUs, or maybe it will do that using some more neuromorphic design, but I really don’t understand why energy density matters very much.
I would suggest reading more about advanced GPU/accelerator design, and then about datacenter design and the thermodynamic/cooling considerations therein.
The vast majority of energy that current humans produce is of course not spent on running human brains, and there are easily 10-30 OOMs of improvement lying around without going into density (just using the energy output of a single power plant under your model would produce something that would likely be easily capable of disempowering humanity).
This is so wildly ridiculous that you really need to show your work. I have already shown some calculations in these threads, but I’ll quickly review here.
A quick google search indicates 1GW is a typical power plant output, which in theory could power roughly a million GPU datacenter. This is almost 100 times larger in power consumption than the current largest official supercomputer: Frontier—which has about 30k GPUs. The supercomputer used to train GPT4 is somewhat of a secret, but estimated to be about that size. So at 50x to 100x you are talking about scaling up to something approaching a hypothetical GPT-5 scale cluster.
Nvidia currently produces less than 100k high end enterprise GPUs per year in total, so you can’t even produce this datacenter unless Nvidia grows by about 10x and TSMC grows by perhaps 2x.
The datacenter would likely cost over a hundred billion dollars, and the resulting models would be proportionally more expensive to run, such that it’s unclear whether this would be a win (at least using current tech). Sure I do think there is some room for software improvement.
But no, I do not think that this hypothetical not currently achievable GPT5 - even if you were running 100k instances of it—would “likely be easily capable of disempowering humanity”.
Of course if we talk longer term, the brain is obviously evidence that one human-brain power can be achieved in about 10 watts, so the 1GW power plant could support a population of 100 million uploads or neuromorphic AGIs. That’s very much part of my model (and hansons, and moravecs) - eventually.
Remember this post is all about critiquing EY’s specific doom model which involves fast foom on current hardware through recursive self-improvement.
Having more room at the bottom is just one of a long list of ways to end up with AIs much smarter than humans. Maybe you have rebuttals to all the other ways AIs could end up much smarter than humans
If you have read much of my writings, you should know that I believe its obvious we will end up with AIs much smarter than humans—but mainly because they will run faster using much more power. In fact this prediction has already come to pass in a limited sense—GPT4 was probably trained on over 100 human lifetimes worth of virtual time/data using only about 3 months of physical time, which represents a 10000x time dilation (but thankfully only for training, not for inference).
I am genuinely curious and confused as to what exactly you concretely imagine this supposed ‘superintelligence’ to be, such that is not already the size of a factory, such that you mention “size of a factory” as if that is something actually worth mentioning—at all. Please show at least your first pass fermi estimates for the compute requirements.
Despite your claim to be “genuinely curious and confused,” the overarching tone and content of this bit does not strike me as projecting curiosity or confusion, but instead confident and sharp-toned burden-of-proof-shifting to habryka. That’s merely a stylistic note, not impacting the content of your claims.
It sounds here like you are agreeing with him that you can deal with any limits to ops/mm^3 limits by simply building a bigger computer. It’s therefore hard for me to see why these arguments about efficiency limitations matter very much for AI’s ability to be superintelligent and exhibit superhuman takeover capabilities.
I can see why maybe human brains, being efficient according to certain metrics, might be a useful tool for the AI to keep around, but I don’t see why we ought to feel at all reassured by that. I don’t really want to serve out the end of my days as an AI’s robot.
I would suggest reading more about advanced GPU/accelerator design, and then about datacenter design and the thermodynamic/cooling considerations therein.
Just as a piece of feedback, this sort of comment is not very enlightening, and it doesn’t convince me that you have background knowledge that ought to make me believe you and not habryka. If your aim is to “teach the crowd,” N of 1, but I am not being effectively taught by this bit.
A quick google search indicates 1GW is a typical power plant output, which in theory could power roughly a million GPU datacenter. This is almost 100 times larger in power consumption than the current largest official supercomputer: Frontier—which has about 30k GPUs. The supercomputer used to train GPT4 is somewhat of a secret, but estimated to be about that size. So at 50x to 100x you are talking about scaling up to something approaching a hypothetical GPT-5 scale cluster.
So it takes 1% of a single power plant output to train GPT4. If GPT-4 got put on chips and distributed, wouldn’t it take only a very small amount of power comparatively to actually run it once trained? Why are we talking about training costs rather than primarily about the cost to operate models once they have been trained?
But no, I do not think that this hypothetical not currently achievable GPT5 - even if you were running 100k instances of it—would “likely be easily capable of disempowering humanity”.′
You have output a lot of writings on this subject and I’ve only read a fraction. Do you argue this point somewhere? This seems pretty cruxy. In my model, at this point, doomers are partly worried about what happens during training of larger and larger models, but they’re also worried about what happens when you proliferate many copies of extant models and put them into something like an AutoGPT format, where they can make plans, access real-world resources, and use plugins.
Again, I think trying to focus the conversation on efficiency by itself is not enough. Right now, I feel like you have a very deep understanding of efficiency, but that once the conversation moves away from the topic of efficiency along some metric and into topics like “why can’t AI just throw more resources at expanding itself” or “why can’t AI take over with a bunch of instances of ChatGPT 4 or 5 in an AutoGPT context” or “why should I be happy about ending my days as the AI’s efficient meat-brain robot,” it becomes hard for me to follow your argument or understand why it’s an important “contra Yudkowsky” or “contra central Yudkowsky-appreciating doomers” argument.
It feels more like constructing, I don’t know, a valid critique of some of the weakest parts (maybe?) of Yudkowsky’s entire written ouvre having to do with the topic of efficiency and nanobots, and then saying that because this fails, the entire Yudkowskian argument about AI doom fails, or even that arguments for AI takeover scenarios fail generally. And I am not at all convinced you’ve shown that it’s on the foundation of Yudkowsky’s (mis?)understanding of efficiency that the rest of his argument stands or falls.
I’ve never feared grey goo or diamondoid bacteria. Mainly, I worry that a malevolent AI might not even need very much intelligence to steamroll humanity. It may just need a combination of the willingness to do damage, the ability to be smarter than a moderately intelligent person in order to hack our social systems, and the ability to proliferate across our computing infrastructure while not caring about what happens to “itself” should individual instances get “caught” and deleted. So much of our ability to mitigate terrorism depends on the ability to catch and punish offenders, the idea that perpetrators care about their own skin and are confined within it. It is the alienness of the form of the AI, and its ability to perform on par with or exceed performance of humans in important areas, to expand the computing and material resources available to itself, that make it seem dangerous to me.
Despite your claim to be “genuinely curious and confused,” the overarching tone and content of this bit does not strike me as projecting curiosity or confusion, but instead confident and sharp-toned burden-of-proof-shifting to habryka
I see how that tone could come off as rude, but really I don’t understand habryka’s model when he says “a superintelligence will just make a brain the size of a factory, and then be in a position to outcompete or destroy humanity quite easily.”
So it takes 1% of a single power plant output to train GPT4. If GPT-4 got put on chips and distributed, wouldn’t it take only a very small amount of power comparatively to actually run it once trained? Why are we talking about training costs rather than primarily about the cost to operate models once they have been trained?
The transformer arch is fully parallelizable only during training, but is roughly just as or more inefficient than RNNs on GPUs/accelerators for inference. The inference costs of GPT4 are of course a openai/microsoft secret, but it is not a cheap model. Also human-level AGI, let alone superintelligence, will likely require continual learning/training.
I guess by “put on chips” you mean baking GPT-4 into an ASIC? That usually doesn’t make sense for fast changing tech, but as moore’s law slows we could start seeing that. I would expect other various changes to tensorcores well before that.
It feels more like constructing, I don’t know, a valid critique of some of the weakest parts (maybe?) of Yudkowsky’s entire written ouvre having to do with the topic of efficiency and nanobots, and then saying that because this fails, the entire Yudkowskian argument about AI doom fails, or even that arguments for AI takeover scenarios fail generally.
The entire specific Yudkowskian argument about hard takeoff via recursive self improving AGI fooming to nanotech is specifically what i’m arguing against, and indeed I only need to argue against the weakest links.
It is the alienness of the form of the AI
See the section on That Alien Mindspace. DL-based AGI is anthropomorphic (as I predicted), not alien.
Re Yudkowsky, I don’t think his entire argument rests on efficiency, and the pieces that don’t can’t be dispatched by arguing about efficiency.
Regarding “alien mindspace,” what I mean is that the physical form of AI, and whatever awareness the AI has of that, makes it alien. Like, if I knew I could potentially transmit my consciousness with perfect precision over the internet and create self-clones almost effortlessly, I would think very differently than I do now.
His argument entirely depends on efficiency. He claims that near future AGI somewhat smarter than us creates even smarter AGI and so on, recursively bottoming out in something that is many many OOM more intelligent than us without using unrealistic amounts of energy, and all of this happens very quickly.
So that’s entirely an argument that boils down to practical computational engineering efficiency considerations. Additionally he needs the AGI to be unaligned by default, and that argument is also faulty.
EY’s model requires slightly-smarter-than-us AGI running on normal hardware to start a FOOM cycle of recursive self improvement resulting in many OOM intelligence improvement in a short amount of time. That requires some combination of 1.) many OOM software improvement on current hardware, 2.) many OOM hardware improvement with current foundry tech, or 3.) completely new foundry tech with many OOM improvement over current—ie nanotech woo. The viability of all/any of this is all entirely dependent on near term engineering practicality.
It seems like in one place, you’re saying EY’s model depends on near term engineering practicality, and in another, that it depends on physics-constrainted efficiency which you argue invalidates it. Being no expert on the physics-based efficiency arguments, I’m happy to concede the physics constraints. But I’m struggling to understand their relevance to non-physics-based efficiency arguments or their strong bearing on matters of engineering practicality.
My understanding is that your argument goes something like this:
You can’t build something many OOMs more intelligent than a brain on hardware with roughly the same size and energy consumption as the brain.
Therefore, building a superintelligent AI would require investing more energy and more material resources than a brain uses.
Therefore… and here’s where the argument loses steam for me. Why can’t we or the AI just invest lots of material and energy resources? How much smarter than us does an unaligned AI need to be to pose a threat, and why should we think resources are a major constraint to get it to recursively self-improve itself to get to that point? Why should we think it will need constant retraining to recursively self-improve? Why do we think it’ll want to keep an economy going?
As far as the “anthropomorphic” counterargument to the “vast space of alien minds” thing, I fully agree that it appears the easiest way to predict tokens from human text is to simulate a human mind. That doesn’t mean the AI is a human mind, or that it is intrinsically constrained to human values. Being able to articulate those values and imitate behaviors that accord with those values is a capability, not a constraint. We have evidence from things like ChaosGPT or jailbreaks that you can easily have the AI behave in ways that appear unaligned, and that even the appearance of consistent alignment has to be consistently enforced in ways that look awfully fragile.
Overall, my sense is that you’ve admirably spent a lot of time probing the physical limits of certain efficiency metrics and how they bear on AI, and I think you have some intriguing arguments about nanotech and “mindspace” and practical engineering as well.
However, I think your arguments would be more impactful if you carefully and consistently delineated these different arguments and attached them more precisely to the EY claims you’re rebutting, and did more work to show how EY’s conclusion X flows from EY’s argument A, and that A is wrong for efficiency reason B, which overturns X but not Y; you disagree with Y for reason C, overturning EY’s argument D. Right now, I think you do make many of these argumentative moves, but they’re sort of scattered across various posts and comments, and I’m open to the idea that they’re all there but I’ve also seen enough inconsistencies to worry that they’re not. To be clear, I would absolutely LOVE it if EY did the very same thing—the burden of proof should ideally not be all on you, and I maintain uncertainty about this whole issue because of the fragmented nature of the debate.
So at this point, it’s hard for me to update more than “some arguments about efficiency and mindspace and practical engineering and nanotech are big points of contention between Jacob and Eliezer.” I’d like to go further and, with you reject arguments that you believe to be false, but I’m not able to do that yet because of the issue that I’m describing here. While I’m hesitant to burden you with additional work, I don’t have the background or the familiarity with your previous writings to do this very effectively—at the end of the day, if anybody’s going to bring together your argument all in one place and make it crystal clear, I think that person has to be you.
His argument entirely depends on efficiency. He claims that near future AGI somewhat smarter than us creates even smarter AGI and so on, recursively bottoming out in something that is many many OOM more intelligent than us without using unrealistic amounts of energy, and all of this happens very quickly.
You just said in your comment to me that a single power plant is enough to run 100M brains. It seems like you need zero hardware progress in order to get something much smarter without unrealistic amounts of energy, so I just don’t understand the relevance of this.
I said longer term—using hypothetical brain-parity neuromorphic computing (uploads or neuromorphic AGI). We need enormous hardware progress to reach that.
Current tech on GPUs requires large supercomputers to train 1e25+ flops models like GPT4 that are approaching, but not quite, human level AGI. If the rurmour of 1T params is true, then it takes a small cluster and ~10KW just to run some smallish number of instances of the model.
Getting something much much smarter than us would require enormous amounts of computation and energy without large advances in software and hardware.
I said longer term—using hypothetical brain-parity neuromorphic computing (uploads or neuromorphic AGI). We need enormous hardware progress to reach that.
Sure. We will probably get enormous hardware progress over the next few decades, so that’s not really an obstacle.
It seems to me your argument is “smarter than human intelligence cannot make enormous hardware or software progress in a relatively short amount of time”, but this has nothing to do with “efficiency arguments”. The bottleneck is not energy, the bottleneck is algorithmic improvements and improvements to GPU production, neither of which is remotely bottlenecked on energy consumption.
Getting something much much smarter than us would require enormous amounts [...] energy without large advances in software and hardware.
No, as you said, it would require like, a power plant worth of energy. Maybe even like 10 power plants or so if you are really stretching it, but as you said, the really central bottleneck here is GPU production, not energy in any relevant way.
Sure. We will probably get enormous hardware progress over the next few decades, so that’s not really an obstacle.
As we get more hardware and slow mostly-aligned AGI/AI progress this further raises the bar for foom.
It seems to me your argument is “smarter than human intelligence cannot make enormous hardware or software progress in a relatively short amount of time”, but this has nothing to do with “efficiency arguments”.
That is actually an efficiency argument, and in my brain efficiency post I discuss multiple sub components of net efficiency that translate into intelligence/$.
The bottleneck is not energy, the bottleneck is algorithmic improvements and improvements to GPU production, neither of which is remotely bottlenecked on energy consumption.
Ahh I see—energy efficiency is tightly coupled to other circuit efficiency metrics as they are all primarily driven by shrinkage. As you increasingly bottom out hardware improvements energy then becomes an increasingly more direct constraint. This is already happening with GPUs where power consumption is roughly doubling with each generation, and could soon dominate operating costs.
See here where I line the roodman model up to future energy usage predictions.
All that being said I do agree that yes the primary bottlneck or crux for the EY fast takeoff/takeover seems to be the amount of slack in software and scaling laws. But only after we agree that there isn’t obvious easy routes for the AGI to bootstrap nanotech assemblers with many OOM greater compute per J than brains or current computers.
Of course if we talk longer term, the brain is obviously evidence that one human-brain power can be achieved in about 10 watts, so the 1GW power plant could support a population of 100 million uploads or neuromorphic AGIs. That’s very much part of my model (and hansons, and moravecs) - eventually.
This seems like it straightforwardly agrees that energy efficiency is not in any way a bottleneck, so I don’t understand the focus of this post on efficiency.
I also don’t know what you mean by longer term. More room at the bottom was of course also talking longer term (you can’t build new hardware in a few weeks, unless you have nanotech, but then you can also build new factories in a few weeks), so I don’t understand why you are suddenly talking as if “longer term” was some kind of shift of the topic.
Eliezer’s model is that we definitely won’t have many decades with AIs smarter but not much smarter than humans, since there appear to be many ways to scale up intelligence, both via algorithmic progress and via hardware progress. Eliezer thinks that drexlerian nanotech is one of the main ways to do this, and if you buy that premise, then the efficiency arguments don’t really matter, since clearly you can just scale things up horizontally and build a bunch of GPUs. But even if you don’t, you can still just scale things up horizontally and increase GPU production (and in any case, energy efficiency is not the bottleneck here, it’s GPU production, which this post doesn’t talk about)
A quick google search indicates 1GW is a typical power plant output, which in theory could power roughly a million GPU datacenter. This is almost 100 times larger in power consumption than the current largest official supercomputer: Frontier—which has about 30k GPUs. The supercomputer used to train GPT4 is somewhat of a secret, but estimated to be about that size. So at 50x to 100x you are talking about scaling up to something approaching a hypothetical GPT-5 scale cluster.
Nvidia currently produces less than 100k high end enterprise GPUs per year in total, so you can’t even produce this datacenter unless Nvidia grows by about 10x and TSMC grows by perhaps 2x.
The datacenter would likely cost over a hundred billion dollars, and the resulting models would be proportionally more expensive to run, such that it’s unclear whether this would be a win (at least using current tech). Sure I do think there is some room for software improvement.
I don’t understand the relevance of this. You seem to be now talking about a completely different scenario than what I understood Eliezer to be talking about. Eliezer does not think that a slightly superhuman AI would be capable of improving the hardware efficiency of its hardware completely on its own.
Both scenarios (going both big, in that you just use whole power-plant levels of energy, or going down in that you improve efficiency of chips) require changing semiconductor manufacturing, which is unlikely to be one of the first things a nascent AI does, unless it does successfully develop and deploy drexlerian nanotech. Eliezer in his model here was talking about what are reasonable limits that we would be approaching here relatively soon after an AI passes human levels.
Remember this post is all about critiquing EY’s specific doom model which involves fast foom on current hardware through recursive self-improvement.
I don’t understand the relevance of thermodynamic efficiency to a foom scenario “on current hardware”. You are not going to change the thermodynamic efficiency of the hardware you are literally running on, you have to build new hardware for that either way.
To reiterate the model of EY that I am critiquing is one where an AGI quickly rapidly fooms through many OOM efficiency improvements. All key required improvements are efficiency improvements—it needs to improve it’s world modelling/planning per unit compute, and or improve compute per dollar and or compute per joule, etc.
In EY’s model there are some perhaps many OOM software improvements over the initial NN arch/aglorithms, perhaps then continued with more OOM hardware improvements. I don’t believe “buying more GPUs” is a key part of his model—it is far far too slow to provide even one OOM upgrade. Renting/hacking your way to even one OOM more GPUs is also largely unrealistic (I run one of the larger GPU compute markets and talk to many suppliers, I have inside knowledge here).
Both scenarios (going both big, in that you just use whole power-plant levels of energy, or going down in that you improve efficiency of chips) require changing semiconductor manufacturing, which is unlikely to be one of the first things a nascent AI does, unless it does successfully develop and deploy drexlerian nanotech
Right, so I have arguments against drexlerian nanotech (Moore room at the bottom, but also the thermodynamic constraints indicating you just can’t get many from nanotech alone), and separate arguments against many OOM from software (mind software efficiency).
I don’t understand the relevance of thermodynamic efficiency to a foom scenario “on current hardware”.
It is mostly relevant to the drexlerian nanotech, as it shows there likely isn’t much improvement over GPUs for all the enormous effort. If nanotech were feasible and could easily allow computers 6 OOM more efficient than the brain using about the same energy/space/materials, then I would more agree with his argument.
I am genuinely curious and confused as to what exactly you concretely imagine this supposed ‘superintelligence’ to be, such that is not already the size of a factory, such that you mention “size of a factory” as if that is something actually worth mentioning—at all. Please show at least your first pass fermi estimates for the compute requirements. By that I mean—what are the compute requirements for the initial SI—and then the later presumably more powerful ‘factory’?
I would suggest reading more about advanced GPU/accelerator design, and then about datacenter design and the thermodynamic/cooling considerations therein.
This is so wildly ridiculous that you really need to show your work. I have already shown some calculations in these threads, but I’ll quickly review here.
A quick google search indicates 1GW is a typical power plant output, which in theory could power roughly a million GPU datacenter. This is almost 100 times larger in power consumption than the current largest official supercomputer: Frontier—which has about 30k GPUs. The supercomputer used to train GPT4 is somewhat of a secret, but estimated to be about that size. So at 50x to 100x you are talking about scaling up to something approaching a hypothetical GPT-5 scale cluster.
Nvidia currently produces less than 100k high end enterprise GPUs per year in total, so you can’t even produce this datacenter unless Nvidia grows by about 10x and TSMC grows by perhaps 2x.
The datacenter would likely cost over a hundred billion dollars, and the resulting models would be proportionally more expensive to run, such that it’s unclear whether this would be a win (at least using current tech). Sure I do think there is some room for software improvement.
But no, I do not think that this hypothetical not currently achievable GPT5 - even if you were running 100k instances of it—would “likely be easily capable of disempowering humanity”.
Of course if we talk longer term, the brain is obviously evidence that one human-brain power can be achieved in about 10 watts, so the 1GW power plant could support a population of 100 million uploads or neuromorphic AGIs. That’s very much part of my model (and hansons, and moravecs) - eventually.
Remember this post is all about critiquing EY’s specific doom model which involves fast foom on current hardware through recursive self-improvement.
If you have read much of my writings, you should know that I believe its obvious we will end up with AIs much smarter than humans—but mainly because they will run faster using much more power. In fact this prediction has already come to pass in a limited sense—GPT4 was probably trained on over 100 human lifetimes worth of virtual time/data using only about 3 months of physical time, which represents a 10000x time dilation (but thankfully only for training, not for inference).
Despite your claim to be “genuinely curious and confused,” the overarching tone and content of this bit does not strike me as projecting curiosity or confusion, but instead confident and sharp-toned burden-of-proof-shifting to habryka. That’s merely a stylistic note, not impacting the content of your claims.
It sounds here like you are agreeing with him that you can deal with any limits to ops/mm^3 limits by simply building a bigger computer. It’s therefore hard for me to see why these arguments about efficiency limitations matter very much for AI’s ability to be superintelligent and exhibit superhuman takeover capabilities.
I can see why maybe human brains, being efficient according to certain metrics, might be a useful tool for the AI to keep around, but I don’t see why we ought to feel at all reassured by that. I don’t really want to serve out the end of my days as an AI’s robot.
Just as a piece of feedback, this sort of comment is not very enlightening, and it doesn’t convince me that you have background knowledge that ought to make me believe you and not habryka. If your aim is to “teach the crowd,” N of 1, but I am not being effectively taught by this bit.
So it takes 1% of a single power plant output to train GPT4. If GPT-4 got put on chips and distributed, wouldn’t it take only a very small amount of power comparatively to actually run it once trained? Why are we talking about training costs rather than primarily about the cost to operate models once they have been trained?
You have output a lot of writings on this subject and I’ve only read a fraction. Do you argue this point somewhere? This seems pretty cruxy. In my model, at this point, doomers are partly worried about what happens during training of larger and larger models, but they’re also worried about what happens when you proliferate many copies of extant models and put them into something like an AutoGPT format, where they can make plans, access real-world resources, and use plugins.
Again, I think trying to focus the conversation on efficiency by itself is not enough. Right now, I feel like you have a very deep understanding of efficiency, but that once the conversation moves away from the topic of efficiency along some metric and into topics like “why can’t AI just throw more resources at expanding itself” or “why can’t AI take over with a bunch of instances of ChatGPT 4 or 5 in an AutoGPT context” or “why should I be happy about ending my days as the AI’s efficient meat-brain robot,” it becomes hard for me to follow your argument or understand why it’s an important “contra Yudkowsky” or “contra central Yudkowsky-appreciating doomers” argument.
It feels more like constructing, I don’t know, a valid critique of some of the weakest parts (maybe?) of Yudkowsky’s entire written ouvre having to do with the topic of efficiency and nanobots, and then saying that because this fails, the entire Yudkowskian argument about AI doom fails, or even that arguments for AI takeover scenarios fail generally. And I am not at all convinced you’ve shown that it’s on the foundation of Yudkowsky’s (mis?)understanding of efficiency that the rest of his argument stands or falls.
I’ve never feared grey goo or diamondoid bacteria. Mainly, I worry that a malevolent AI might not even need very much intelligence to steamroll humanity. It may just need a combination of the willingness to do damage, the ability to be smarter than a moderately intelligent person in order to hack our social systems, and the ability to proliferate across our computing infrastructure while not caring about what happens to “itself” should individual instances get “caught” and deleted. So much of our ability to mitigate terrorism depends on the ability to catch and punish offenders, the idea that perpetrators care about their own skin and are confined within it. It is the alienness of the form of the AI, and its ability to perform on par with or exceed performance of humans in important areas, to expand the computing and material resources available to itself, that make it seem dangerous to me.
I see how that tone could come off as rude, but really I don’t understand habryka’s model when he says “a superintelligence will just make a brain the size of a factory, and then be in a position to outcompete or destroy humanity quite easily.”
The transformer arch is fully parallelizable only during training, but is roughly just as or more inefficient than RNNs on GPUs/accelerators for inference. The inference costs of GPT4 are of course a openai/microsoft secret, but it is not a cheap model. Also human-level AGI, let alone superintelligence, will likely require continual learning/training.
I guess by “put on chips” you mean baking GPT-4 into an ASIC? That usually doesn’t make sense for fast changing tech, but as moore’s law slows we could start seeing that. I would expect other various changes to tensorcores well before that.
The entire specific Yudkowskian argument about hard takeoff via recursive self improving AGI fooming to nanotech is specifically what i’m arguing against, and indeed I only need to argue against the weakest links.
See the section on That Alien Mindspace. DL-based AGI is anthropomorphic (as I predicted), not alien.
Re Yudkowsky, I don’t think his entire argument rests on efficiency, and the pieces that don’t can’t be dispatched by arguing about efficiency.
Regarding “alien mindspace,” what I mean is that the physical form of AI, and whatever awareness the AI has of that, makes it alien. Like, if I knew I could potentially transmit my consciousness with perfect precision over the internet and create self-clones almost effortlessly, I would think very differently than I do now.
His argument entirely depends on efficiency. He claims that near future AGI somewhat smarter than us creates even smarter AGI and so on, recursively bottoming out in something that is many many OOM more intelligent than us without using unrealistic amounts of energy, and all of this happens very quickly.
So that’s entirely an argument that boils down to practical computational engineering efficiency considerations. Additionally he needs the AGI to be unaligned by default, and that argument is also faulty.
In your other recent comment to me, you said:
It seems like in one place, you’re saying EY’s model depends on near term engineering practicality, and in another, that it depends on physics-constrainted efficiency which you argue invalidates it. Being no expert on the physics-based efficiency arguments, I’m happy to concede the physics constraints. But I’m struggling to understand their relevance to non-physics-based efficiency arguments or their strong bearing on matters of engineering practicality.
My understanding is that your argument goes something like this:
You can’t build something many OOMs more intelligent than a brain on hardware with roughly the same size and energy consumption as the brain.
Therefore, building a superintelligent AI would require investing more energy and more material resources than a brain uses.
Therefore… and here’s where the argument loses steam for me. Why can’t we or the AI just invest lots of material and energy resources? How much smarter than us does an unaligned AI need to be to pose a threat, and why should we think resources are a major constraint to get it to recursively self-improve itself to get to that point? Why should we think it will need constant retraining to recursively self-improve? Why do we think it’ll want to keep an economy going?
As far as the “anthropomorphic” counterargument to the “vast space of alien minds” thing, I fully agree that it appears the easiest way to predict tokens from human text is to simulate a human mind. That doesn’t mean the AI is a human mind, or that it is intrinsically constrained to human values. Being able to articulate those values and imitate behaviors that accord with those values is a capability, not a constraint. We have evidence from things like ChaosGPT or jailbreaks that you can easily have the AI behave in ways that appear unaligned, and that even the appearance of consistent alignment has to be consistently enforced in ways that look awfully fragile.
Overall, my sense is that you’ve admirably spent a lot of time probing the physical limits of certain efficiency metrics and how they bear on AI, and I think you have some intriguing arguments about nanotech and “mindspace” and practical engineering as well.
However, I think your arguments would be more impactful if you carefully and consistently delineated these different arguments and attached them more precisely to the EY claims you’re rebutting, and did more work to show how EY’s conclusion X flows from EY’s argument A, and that A is wrong for efficiency reason B, which overturns X but not Y; you disagree with Y for reason C, overturning EY’s argument D. Right now, I think you do make many of these argumentative moves, but they’re sort of scattered across various posts and comments, and I’m open to the idea that they’re all there but I’ve also seen enough inconsistencies to worry that they’re not. To be clear, I would absolutely LOVE it if EY did the very same thing—the burden of proof should ideally not be all on you, and I maintain uncertainty about this whole issue because of the fragmented nature of the debate.
So at this point, it’s hard for me to update more than “some arguments about efficiency and mindspace and practical engineering and nanotech are big points of contention between Jacob and Eliezer.” I’d like to go further and, with you reject arguments that you believe to be false, but I’m not able to do that yet because of the issue that I’m describing here. While I’m hesitant to burden you with additional work, I don’t have the background or the familiarity with your previous writings to do this very effectively—at the end of the day, if anybody’s going to bring together your argument all in one place and make it crystal clear, I think that person has to be you.
You just said in your comment to me that a single power plant is enough to run 100M brains. It seems like you need zero hardware progress in order to get something much smarter without unrealistic amounts of energy, so I just don’t understand the relevance of this.
I said longer term—using hypothetical brain-parity neuromorphic computing (uploads or neuromorphic AGI). We need enormous hardware progress to reach that.
Current tech on GPUs requires large supercomputers to train 1e25+ flops models like GPT4 that are approaching, but not quite, human level AGI. If the rurmour of 1T params is true, then it takes a small cluster and ~10KW just to run some smallish number of instances of the model.
Getting something much much smarter than us would require enormous amounts of computation and energy without large advances in software and hardware.
Sure. We will probably get enormous hardware progress over the next few decades, so that’s not really an obstacle.
It seems to me your argument is “smarter than human intelligence cannot make enormous hardware or software progress in a relatively short amount of time”, but this has nothing to do with “efficiency arguments”. The bottleneck is not energy, the bottleneck is algorithmic improvements and improvements to GPU production, neither of which is remotely bottlenecked on energy consumption.
No, as you said, it would require like, a power plant worth of energy. Maybe even like 10 power plants or so if you are really stretching it, but as you said, the really central bottleneck here is GPU production, not energy in any relevant way.
As we get more hardware and slow mostly-aligned AGI/AI progress this further raises the bar for foom.
That is actually an efficiency argument, and in my brain efficiency post I discuss multiple sub components of net efficiency that translate into intelligence/$.
Ahh I see—energy efficiency is tightly coupled to other circuit efficiency metrics as they are all primarily driven by shrinkage. As you increasingly bottom out hardware improvements energy then becomes an increasingly more direct constraint. This is already happening with GPUs where power consumption is roughly doubling with each generation, and could soon dominate operating costs.
See here where I line the roodman model up to future energy usage predictions.
All that being said I do agree that yes the primary bottlneck or crux for the EY fast takeoff/takeover seems to be the amount of slack in software and scaling laws. But only after we agree that there isn’t obvious easy routes for the AGI to bootstrap nanotech assemblers with many OOM greater compute per J than brains or current computers.
How much room is there in algorithmic improvements?
Maybe it would be a good idea to change the title of this essay to:
as to not give people hope that there would be a counter argument somewhere in this article to his more general claim:
This seems like it straightforwardly agrees that energy efficiency is not in any way a bottleneck, so I don’t understand the focus of this post on efficiency.
I also don’t know what you mean by longer term. More room at the bottom was of course also talking longer term (you can’t build new hardware in a few weeks, unless you have nanotech, but then you can also build new factories in a few weeks), so I don’t understand why you are suddenly talking as if “longer term” was some kind of shift of the topic.
Eliezer’s model is that we definitely won’t have many decades with AIs smarter but not much smarter than humans, since there appear to be many ways to scale up intelligence, both via algorithmic progress and via hardware progress. Eliezer thinks that drexlerian nanotech is one of the main ways to do this, and if you buy that premise, then the efficiency arguments don’t really matter, since clearly you can just scale things up horizontally and build a bunch of GPUs. But even if you don’t, you can still just scale things up horizontally and increase GPU production (and in any case, energy efficiency is not the bottleneck here, it’s GPU production, which this post doesn’t talk about)
I don’t understand the relevance of this. You seem to be now talking about a completely different scenario than what I understood Eliezer to be talking about. Eliezer does not think that a slightly superhuman AI would be capable of improving the hardware efficiency of its hardware completely on its own.
Both scenarios (going both big, in that you just use whole power-plant levels of energy, or going down in that you improve efficiency of chips) require changing semiconductor manufacturing, which is unlikely to be one of the first things a nascent AI does, unless it does successfully develop and deploy drexlerian nanotech. Eliezer in his model here was talking about what are reasonable limits that we would be approaching here relatively soon after an AI passes human levels.
I don’t understand the relevance of thermodynamic efficiency to a foom scenario “on current hardware”. You are not going to change the thermodynamic efficiency of the hardware you are literally running on, you have to build new hardware for that either way.
To reiterate the model of EY that I am critiquing is one where an AGI quickly rapidly fooms through many OOM efficiency improvements. All key required improvements are efficiency improvements—it needs to improve it’s world modelling/planning per unit compute, and or improve compute per dollar and or compute per joule, etc.
In EY’s model there are some perhaps many OOM software improvements over the initial NN arch/aglorithms, perhaps then continued with more OOM hardware improvements. I don’t believe “buying more GPUs” is a key part of his model—it is far far too slow to provide even one OOM upgrade. Renting/hacking your way to even one OOM more GPUs is also largely unrealistic (I run one of the larger GPU compute markets and talk to many suppliers, I have inside knowledge here).
Right, so I have arguments against drexlerian nanotech (Moore room at the bottom, but also the thermodynamic constraints indicating you just can’t get many from nanotech alone), and separate arguments against many OOM from software (mind software efficiency).
It is mostly relevant to the drexlerian nanotech, as it shows there likely isn’t much improvement over GPUs for all the enormous effort. If nanotech were feasible and could easily allow computers 6 OOM more efficient than the brain using about the same energy/space/materials, then I would more agree with his argument.