I feel like even under the worldview that your beliefs imply, a superintelligence will just make a brain the size of a factory, and then be in a position to outcompete or destroy humanity quite easily.
The brain is a million times slower than digital computers, but its slow speed is probably efficient for its given energy budget, as it allows for a full utilization of an enormous memory capacity and memory bandwidth. As a consequence of being very slow, brains are enormously circuit cycle efficient. Thus even some hypothetical superintelligence, running on non-exotic hardware, will not be able to think much faster than an artificial brain running on equivalent hardware at the same clock rate.
Let’s accept all Jacob’s analysis about the tradeoffs of clock speed, memory capacity and bandwidth.
The force of his conclusion depends on the superintelligence “running on equivalent hardware.” Obviously, core to Eliezer’s superintelligence argument, and habryka’s comment here, is the point that the hardware underpinning AI can be made large and expanded upon in a way that is not possible for human brains.
Jacob knows this, and addresses it in comments in response to Vaniver pointing out that birds may be more efficient than jet planes in terms of calories/mile flown, but that when the relevant metric is top speed or human passengers carried, the jet wins. Jacob responds:
I agree electricity is cheap, and discuss that. But electricity is not free, and still becomes a constraint...
The rental price using enterprise GPUs is at least 4x as much, so more like $20,000/yr per agent. So the potential economic advantage is not yet multiple OOM. It’s actually more like little to no advantage for low-end robotic labor, or perhaps 1 OOM advantage for programmers/researchers/ec. But if we had AGI today GPU prices would just skyrocket to arbitrage that advantage, at least until foundries could ramp up GPU production.
So the crux here appears to be about the practicality of replacing human brains with factory-sized artificial ones, in terms of physical resource limitations.
Daniel Kokotajlo disagrees that this is important:
$2,000/yr per agent is nothing, when we are talking about hypothetical AGI. This seems to be evidence against your claim that energy is a taut constraint.
Jacob doubles down that it is:
Energy is always an engineering constraint: it’s a primary constraint on Moore’s Law, and thus also a primary limiter on a fast takeoff with GPUs (because world power supply isn’t enough to support net ANN compute much larger than current brain population net compute).
But again I already indicated it’s probably not a ‘taut constraint’ on early AGI in terms of economic cost—at least in my model of likely requirements for early not-smarter-than-human AGI.
Also yes additionally longer term we can expect energy to become a larger fraction of economic cost—through some combination of more efficient chip production, or just the slowing of moore’s law itself (which implies chips holding value for much longer, thus reducing the dominant hardware depreciation component of rental costs)
So Jacob here admits that energy is neither a ‘taut constraint’ for early AGI, and that at the same time it will be a larger fraction of the cost. In other words, it’s not a bottleneck for AGI, and no other resource is either.
This is where Jacob’s discussion ended.
So I think Jacob has at least two jobs to do to convince me. I would be very pleased and appreciative if he achieved just one of them.
First, he needs to explain why any efficiency constraints can’t be overcome by just throwing a lot of material and energy resources into building and powering inefficient or as-efficient-as-human-brains GPUs. If energy is not a taut constraint for AGI, and it’s also expected to be an increasing fraction of costs over time, then that sounds like an argument that we can overcome any efficiency limits with increasing expenditures to achieve superhuman performance.
Second, he needs to explain why things like energy, size, or ops/sec efficiency are the most important efficiency metrics as opposed to things like “physical tasks/second,” or “brain-size intelligences produced per year,” or “speed at which information can be taken in and processed via sensors positioned around the globe.” There are so very many efficiency (“useful output/resource input”) metrics that we can construct, and on many of them, the human brain and body are demonstrably nowhere near the physical limit.
Right now, doubling down on physics-based efficiency arguments, as he’s doing here, don’t feel like a winning strategy to me.
First, he needs to explain why any efficiency constraints can’t be overcome by just throwing a lot of material and energy resources into building and powering inefficient or as-efficient-as-human-brains GPUs. If energy is not a taut constraint for AGI, and it’s also expected to be an increasing fraction of costs over time, then that sounds like an argument that we can overcome any efficiency limits with increasing expenditures to achieve superhuman performance.
If Jake claims to disagree with the claim that ai can starkly surpass humans [now disproven—he has made more explicit that it can], I’d roll my eyes at him. He is doing a significant amount of work based on the premise that this ai can surpass humans. His claims about safety must therefore not rely on ai being limited in capability; if his claims had relied on ai being naturally capability bounded I’d have rolled to disbelieve [edit: his claims do not rely on it]. I don’t think his claims rely on it, as I currently think his views on safety are damn close to simply being a lower resolution version of mine held overconfidently [this is intended to be a pointer to stalking both our profiles]; it’s possible he actually disagrees with my views, but so far my impression is he has some really good overall ideas but hasn’t thought in detail about how to mitigate the problems I see. But I have almost always agreed with him about the rest of the points he explicitly spells out in OP, with some exceptions where he had to talk me into his view and I eventually became convinced. (I really doubted the energy cost of the brain being near optimal for energy budget and temperature target. I later came to realize it being near optimal is fundamental to why it works at all.)
from what he’s told me and what I’ve seen him say, my impression is he hasn’t looked quite as closely at safety as I have, and to be clear, I don’t think either of us are proper experts on co-protective systems alignment or open source game theory or any of that fancy high end alignment stuff; I worked with him first while I was initially studying machine learning 2015-2016, then we worked together on a research project which then pivoted to building vast.ai. I’ve since moved on to more studying, but given assumption of our otherwise mostly shared background assumptions with varying levels of skill (read: he’s still much more skilled on some core fundamentals and I’ve settled into just being a nerd who likes to read interesting papers), I think our views are still mostly shared to the degree our knowledge overlaps.
@ Jake, re: safety, I just wish you had the kind of mind that was habitually allergic to C++’s safety issues and desperate for the safety of rustlang, exactly bounded approximation is great. Of course, we’ve had that discussion many times, he’s quite the template wizard, with all the good and bad that comes with that.
(open source game theory is a kind of template magic)
Respectfully, it’s hard for me to follow your comment because of the amount of times you say things like “If Jake claims to disagree with this,” “based on the premise that this is false,” “must therefore not rely on it or be false,” and “I don’t think they rely on it.” The double negatives plus pointing to things with the word “this” and “it” makes me lose confidence in my ability to track your line of thinking. If you could speak in the positive and replace your “pointer terms” like “this” and “it” with the concrete claims you’re referring to, that would help a lot!
Understandable, I edited in clearer references—did that resolve all the issues? I’m not sure in return that I parsed all your issues parsing :) I appreciate the specific request!
It helps! There are still some double negatives (“His claims about safety must therefore not rely on ai not surpassing humans, or be false” could be reworded to “his claims about safety can only be true if they allow for AI surpassing humans,” for example), and I, not being a superintelligence, would find that easier to parse :)
The “pointers” bit is mostly fixed by you replacing the word “this” with the phrase “the claim that ai can starkly surpass humans.” Thank you for the edits!
First, he needs to explain why any efficiency constraints can’t be overcome by just throwing a lot of material and energy resources into building and powering inefficient or as-efficient-as-human-brains GPUs. If energy is not a taut constraint for AGI, and it’s also expected to be an increasing fraction of costs over time, then that sounds like an argument that we can overcome any efficiency limits with increasing expenditures to achieve superhuman performance.
I don’t need to explain that as I don’t believe it. Of course you can overcome efficiency constraints somewhat by brute force—and that is why I agree energy is not by itself an especially taut constraint for early AGI, but it is a taut constraint for SI.
You can’t overcome any limits just by increasing expenditures. See my reply here for an example.
Second, he needs to explain why things like energy, size, or ops/sec efficiency are the most important efficiency metrics
I don’t really feel this need, because EY already agrees thermodynamic efficiency is important, and i’m arguing specifically against core claims of his model.
Computation simply is energy organized towards some end, and intelligence is a form of computation. A superintelligence that can clearly overpower humanity is—almost by definition—something with greater intelligence than humanity, which thus translates into compute and energy requirements through efficiency factors.
It’s absolutely valid to make a local argument against specific parts of Eliezer’s model. However, you have a lot of other arguments “attached” that don’t straightforwardly flow from the parts of Eliezer’s model you’re mainly attacking. That’s a debate style choice that’s up to you, but as a reader who is hoping to learn from you, it becomes distracting because I have to put a lot of extra work into distinguishing “this is a key argument against point 3 from EY’s efficiency model” from “this is a side argument consisting of one assertion about bioweapons based on unstated biology background knowledge.”
Would it be better if we switched from interpreting your post as “a tightly focused argument on demolishing EY’s core efficiency-based arguments,” to “laying out Jabob’s overall view on AI risk, with a lot of emphasis on efficiency arguments?” If that’s the best way to look at it then I retract the objection I’m making here, except to say it wasn’t as clear as it could have been.
The bioweapons is something of a tangent, but I felled compelled to mention it because every time I’ve pointed out that strong nanotech can’t have any core thermodynamic efficiency over biology someone has to mention superviruses or something, even that isn’t part of EY’s model—he talks about diamond nanobots. But sure, that paragraph is something of a tangent.
EY’s model requires slightly-smarter-than-us AGI running on normal hardware to start a FOOM cycle of recursive self improvement resulting in many OOM intelligence improvement in a short amount of time. That requires some combination of 1.) many OOM software improvement on current hardware, 2.) many OOM hardware improvement with current foundry tech, or 3.) completely new foundry tech with many OOM improvement over current—ie nanotech woo. The viability of all/any of this is all entirely dependent on near term engineering practicality.
I think I see what you’re saying here. Correct me if I’m wrong.
You’re saying that there’s an argument floating around that goes something like this:
At some point in the AI training process, there might be an “awakening” of the AI to an understanding of its situation, its goal, and the state of the world. The AI, while being trained, will realize that to pursue the goal it’s being trained on most effectively, it needs to be a lot smarter and more powerful. Being already superintelligent, it will, during the training process, figure out ways to use existing hardware and energy infrastructure to make itself even more intelligent, without alerting humans. Of course, it can’t build new hardware or noticeably disrupt existing hardware beyond that which has been allocated to it, since that would trigger an investigation and shutdown by humans.
And it’s this argument specifically that you are dispatching with your efficiency arguments. Because, for inescapable physics reasons, AI will hit an efficiency wall, and it can’t become more intelligent than humans on hardware with equivalent size, energy, and so on. Loosely speaking, it’s impossible to build a device something significantly smaller than a brain and using less power than a brain running AI that’s more than 1-2 OOMs smarter than a brain, and we can certainly rule out a superintelligence 6 OOMs smarter than humans running on a device smaller and less energy-intensive than a brain.
You have other arguments about practical engineering constraints, the potential utility to an AI of keeping humans around, the difficulty of building grey goo, and so on, the “alien minds” argument, but those are all based on separate counterarguments. You’re also not arguing about whether an AI just 2-100x as intelligent as humans might be dangerous based on efficiency considerations.
You do have arguments in some or all of these areas, but the efficiency arguments are meant to just deal with this one specific scenario about a 6 OOM (not a 2 OOM) improvement in intelligence during a training run without accessing more hardware than was made available during the training run.
I’m confused because you describe an “argument specifically that you are dispatching with your efficiency arguments”, and the first paragraph sounds like an EY argument, but the 2nd more like my argument. (And ‘dispatching’ is ambiguous)
Also “being already superintelligent” presumes the conclusion at the onset.
So lets restart:
Someone creates an AGI a bit smarter than humans.
It creates even smarter AGI—by rewriting its own source code.
After the Nth iteration and software OOM improvement is tapped it creates nanotech assemblers to continue growing OOM in power (or alternatively somehow gets OOM improvement with existing foundry tech, but that seems less likely as part of EY’s model).
At some point it has more intelligence/compute than all of humanity, and kills us with nanotech or something.
EY and I agree on 1 but diverge past that. Point 2 is partly a matter of software efficiency but not entirely. Recall that I correctly predicted in advance that AGI requires brain-like massive training compute, which largely defeats EY’s view of 2 where it’s just a modest “rewrite of its own source code”. The efficiency considerations matter for both 2 and 3, as they determine how effectively it can quickly turn resources (energy/materials/money/etc) into bigger better training runs to upgrade its intelligence.
I’m confused because you describe an “argument specifically that you are dispatching with your efficiency arguments”, and the first paragraph sounds like an EY argument, but the 2nd more like my argument. (And ‘dispatching’ is ambiguous)
Ugh yes, I have no idea why I originally formatted it with the second paragraph quoted as I had it originally (which I fully intended as an articulation of your argument, a rebuttal to the first EY-style paragraph). Just a confusing formatting and structure error on my part. Sorry about that, thanks for your patience.
So as a summary, you agree that AI could be trained a bit smarter than humans, but you disagree with the model where AI could suddenly iteratively extract like 6 OOMs better performance on the same hardware it’s running on, all at once, figure out ways to interact with the physical world again within the hardware it’s already training on, and then strike humanity all at once with undetectable nanotech before the training run is even complete.
The inability of the AI to attain 6 OOMs better performance on its training hardwareduring its training run by recursively self-improving its own software is mainly based on physical efficiency limits, and this is why you put such heavy emphasis on them. And the idea that neural net-like structures that are very demanding in terms of compute, energy, space, etc appear to be the only tractable road to superintelligence means that there’s no alternative, much more efficient scheme the neural net form of the AI could find to rewrite itself a fundamentally more efficienct architecture on this scale. Again, you have other arguments to deal with other concerns and to make other predictions about the outcome of training superintelligent AI, but dispatching this specific scenario is where your efficiency arguments are most important.
Yes but I again expect AGI to use continuous learning, so the training run doesn’t really end. But yes I largely agree with that summary.
NN/DL in its various flavors are simply what efficient approx bayesian inference involves, and there are not viable non-equivalent dramatically better alternatives.
Thanks Jacob for talking me through your model. I agree with you that this is a model that EY and others associated with him have put forth. I’ve looked back through Eliezer’s old posts, and he is consistently against the idea that LLMs are the path to superintelligence (not just that they’re not the only path, but he outright denies that superintelligence could come from neural nets).
My update, based on your arguments here, is that any future claim about a mechanism for iterative self-improvement that happens suddenly, on the training hardware and involves > 2 OOMs of improvement, needs to first deal with the objections you are raising here to be a meaningful way of moving the conversation forward.
I’m going to expand on this.
Jacob’s conclusion to the speed section of his post on brain efficiency is this:
Let’s accept all Jacob’s analysis about the tradeoffs of clock speed, memory capacity and bandwidth.
The force of his conclusion depends on the superintelligence “running on equivalent hardware.” Obviously, core to Eliezer’s superintelligence argument, and habryka’s comment here, is the point that the hardware underpinning AI can be made large and expanded upon in a way that is not possible for human brains.
Jacob knows this, and addresses it in comments in response to Vaniver pointing out that birds may be more efficient than jet planes in terms of calories/mile flown, but that when the relevant metric is top speed or human passengers carried, the jet wins. Jacob responds:
So the crux here appears to be about the practicality of replacing human brains with factory-sized artificial ones, in terms of physical resource limitations.
Daniel Kokotajlo disagrees that this is important:
Jacob doubles down that it is:
So Jacob here admits that energy is neither a ‘taut constraint’ for early AGI, and that at the same time it will be a larger fraction of the cost. In other words, it’s not a bottleneck for AGI, and no other resource is either.
This is where Jacob’s discussion ended.
So I think Jacob has at least two jobs to do to convince me. I would be very pleased and appreciative if he achieved just one of them.
First, he needs to explain why any efficiency constraints can’t be overcome by just throwing a lot of material and energy resources into building and powering inefficient or as-efficient-as-human-brains GPUs. If energy is not a taut constraint for AGI, and it’s also expected to be an increasing fraction of costs over time, then that sounds like an argument that we can overcome any efficiency limits with increasing expenditures to achieve superhuman performance.
Second, he needs to explain why things like energy, size, or ops/sec efficiency are the most important efficiency metrics as opposed to things like “physical tasks/second,” or “brain-size intelligences produced per year,” or “speed at which information can be taken in and processed via sensors positioned around the globe.” There are so very many efficiency (“useful output/resource input”) metrics that we can construct, and on many of them, the human brain and body are demonstrably nowhere near the physical limit.
Right now, doubling down on physics-based efficiency arguments, as he’s doing here, don’t feel like a winning strategy to me.
If Jake claims to disagree with the claim that ai can starkly surpass humans [now disproven—he has made more explicit that it can], I’d roll my eyes at him. He is doing a significant amount of work based on the premise that this ai can surpass humans. His claims about safety must therefore not rely on ai being limited in capability; if his claims had relied on ai being naturally capability bounded I’d have rolled to disbelieve [edit: his claims do not rely on it]. I don’t think his claims rely on it, as I currently think his views on safety are damn close to simply being a lower resolution version of mine held overconfidently [this is intended to be a pointer to stalking both our profiles]; it’s possible he actually disagrees with my views, but so far my impression is he has some really good overall ideas but hasn’t thought in detail about how to mitigate the problems I see. But I have almost always agreed with him about the rest of the points he explicitly spells out in OP, with some exceptions where he had to talk me into his view and I eventually became convinced. (I really doubted the energy cost of the brain being near optimal for energy budget and temperature target. I later came to realize it being near optimal is fundamental to why it works at all.)
from what he’s told me and what I’ve seen him say, my impression is he hasn’t looked quite as closely at safety as I have, and to be clear, I don’t think either of us are proper experts on co-protective systems alignment or open source game theory or any of that fancy high end alignment stuff; I worked with him first while I was initially studying machine learning 2015-2016, then we worked together on a research project which then pivoted to building vast.ai. I’ve since moved on to more studying, but given assumption of our otherwise mostly shared background assumptions with varying levels of skill (read: he’s still much more skilled on some core fundamentals and I’ve settled into just being a nerd who likes to read interesting papers), I think our views are still mostly shared to the degree our knowledge overlaps.
@ Jake, re: safety, I just wish you had the kind of mind that was habitually allergic to C++’s safety issues and desperate for the safety of rustlang, exactly bounded approximation is great. Of course, we’ve had that discussion many times, he’s quite the template wizard, with all the good and bad that comes with that.
(open source game theory is a kind of template magic)
Respectfully, it’s hard for me to follow your comment because of the amount of times you say things like “If Jake claims to disagree with this,” “based on the premise that this is false,” “must therefore not rely on it or be false,” and “I don’t think they rely on it.” The double negatives plus pointing to things with the word “this” and “it” makes me lose confidence in my ability to track your line of thinking. If you could speak in the positive and replace your “pointer terms” like “this” and “it” with the concrete claims you’re referring to, that would help a lot!
Understandable, I edited in clearer references—did that resolve all the issues? I’m not sure in return that I parsed all your issues parsing :) I appreciate the specific request!
It helps! There are still some double negatives (“His claims about safety must therefore not rely on ai not surpassing humans, or be false” could be reworded to “his claims about safety can only be true if they allow for AI surpassing humans,” for example), and I, not being a superintelligence, would find that easier to parse :)
The “pointers” bit is mostly fixed by you replacing the word “this” with the phrase “the claim that ai can starkly surpass humans.” Thank you for the edits!
I don’t need to explain that as I don’t believe it. Of course you can overcome efficiency constraints somewhat by brute force—and that is why I agree energy is not by itself an especially taut constraint for early AGI, but it is a taut constraint for SI.
You can’t overcome any limits just by increasing expenditures. See my reply here for an example.
I don’t really feel this need, because EY already agrees thermodynamic efficiency is important, and i’m arguing specifically against core claims of his model.
Computation simply is energy organized towards some end, and intelligence is a form of computation. A superintelligence that can clearly overpower humanity is—almost by definition—something with greater intelligence than humanity, which thus translates into compute and energy requirements through efficiency factors.
It’s absolutely valid to make a local argument against specific parts of Eliezer’s model. However, you have a lot of other arguments “attached” that don’t straightforwardly flow from the parts of Eliezer’s model you’re mainly attacking. That’s a debate style choice that’s up to you, but as a reader who is hoping to learn from you, it becomes distracting because I have to put a lot of extra work into distinguishing “this is a key argument against point 3 from EY’s efficiency model” from “this is a side argument consisting of one assertion about bioweapons based on unstated biology background knowledge.”
Would it be better if we switched from interpreting your post as “a tightly focused argument on demolishing EY’s core efficiency-based arguments,” to “laying out Jabob’s overall view on AI risk, with a lot of emphasis on efficiency arguments?” If that’s the best way to look at it then I retract the objection I’m making here, except to say it wasn’t as clear as it could have been.
The bioweapons is something of a tangent, but I felled compelled to mention it because every time I’ve pointed out that strong nanotech can’t have any core thermodynamic efficiency over biology someone has to mention superviruses or something, even that isn’t part of EY’s model—he talks about diamond nanobots. But sure, that paragraph is something of a tangent.
EY’s model requires slightly-smarter-than-us AGI running on normal hardware to start a FOOM cycle of recursive self improvement resulting in many OOM intelligence improvement in a short amount of time. That requires some combination of 1.) many OOM software improvement on current hardware, 2.) many OOM hardware improvement with current foundry tech, or 3.) completely new foundry tech with many OOM improvement over current—ie nanotech woo. The viability of all/any of this is all entirely dependent on near term engineering practicality.
I think I see what you’re saying here. Correct me if I’m wrong.
You’re saying that there’s an argument floating around that goes something like this:
And it’s this argument specifically that you are dispatching with your efficiency arguments. Because, for inescapable physics reasons, AI will hit an efficiency wall, and it can’t become more intelligent than humans on hardware with equivalent size, energy, and so on. Loosely speaking, it’s impossible to build a device something significantly smaller than a brain and using less power than a brain running AI that’s more than 1-2 OOMs smarter than a brain, and we can certainly rule out a superintelligence 6 OOMs smarter than humans running on a device smaller and less energy-intensive than a brain.
You have other arguments about practical engineering constraints, the potential utility to an AI of keeping humans around, the difficulty of building grey goo, and so on, the “alien minds” argument, but those are all based on separate counterarguments. You’re also not arguing about whether an AI just 2-100x as intelligent as humans might be dangerous based on efficiency considerations.
You do have arguments in some or all of these areas, but the efficiency arguments are meant to just deal with this one specific scenario about a 6 OOM (not a 2 OOM) improvement in intelligence during a training run without accessing more hardware than was made available during the training run.
Is that correct?
I’m confused because you describe an “argument specifically that you are dispatching with your efficiency arguments”, and the first paragraph sounds like an EY argument, but the 2nd more like my argument. (And ‘dispatching’ is ambiguous)
Also “being already superintelligent” presumes the conclusion at the onset.
So lets restart:
Someone creates an AGI a bit smarter than humans.
It creates even smarter AGI—by rewriting its own source code.
After the Nth iteration and software OOM improvement is tapped it creates nanotech assemblers to continue growing OOM in power (or alternatively somehow gets OOM improvement with existing foundry tech, but that seems less likely as part of EY’s model).
At some point it has more intelligence/compute than all of humanity, and kills us with nanotech or something.
EY and I agree on 1 but diverge past that. Point 2 is partly a matter of software efficiency but not entirely. Recall that I correctly predicted in advance that AGI requires brain-like massive training compute, which largely defeats EY’s view of 2 where it’s just a modest “rewrite of its own source code”. The efficiency considerations matter for both 2 and 3, as they determine how effectively it can quickly turn resources (energy/materials/money/etc) into bigger better training runs to upgrade its intelligence.
Ugh yes, I have no idea why I originally formatted it with the second paragraph quoted as I had it originally (which I fully intended as an articulation of your argument, a rebuttal to the first EY-style paragraph). Just a confusing formatting and structure error on my part. Sorry about that, thanks for your patience.
So as a summary, you agree that AI could be trained a bit smarter than humans, but you disagree with the model where AI could suddenly iteratively extract like 6 OOMs better performance on the same hardware it’s running on, all at once, figure out ways to interact with the physical world again within the hardware it’s already training on, and then strike humanity all at once with undetectable nanotech before the training run is even complete.
The inability of the AI to attain 6 OOMs better performance on its training hardware during its training run by recursively self-improving its own software is mainly based on physical efficiency limits, and this is why you put such heavy emphasis on them. And the idea that neural net-like structures that are very demanding in terms of compute, energy, space, etc appear to be the only tractable road to superintelligence means that there’s no alternative, much more efficient scheme the neural net form of the AI could find to rewrite itself a fundamentally more efficienct architecture on this scale. Again, you have other arguments to deal with other concerns and to make other predictions about the outcome of training superintelligent AI, but dispatching this specific scenario is where your efficiency arguments are most important.
Is that correct?
Yes but I again expect AGI to use continuous learning, so the training run doesn’t really end. But yes I largely agree with that summary.
NN/DL in its various flavors are simply what efficient approx bayesian inference involves, and there are not viable non-equivalent dramatically better alternatives.
Thanks Jacob for talking me through your model. I agree with you that this is a model that EY and others associated with him have put forth. I’ve looked back through Eliezer’s old posts, and he is consistently against the idea that LLMs are the path to superintelligence (not just that they’re not the only path, but he outright denies that superintelligence could come from neural nets).
My update, based on your arguments here, is that any future claim about a mechanism for iterative self-improvement that happens suddenly, on the training hardware and involves > 2 OOMs of improvement, needs to first deal with the objections you are raising here to be a meaningful way of moving the conversation forward.