Thanks for facilitating this! I found Steven’s post and spxtr’s comments particularly insightful.
I still think there’s a more fundamental issue with Jacob’s analysis, which is also present in some other works that characterize the brain as a substrate for various kinds of computation usually performed in silicon.
Namely, Jacob (and others) are implicitly or explicitly comparing FLOPS with “synaptic ops”, but these quantities are fundamentally incomparable. (I’ve been meaning to turn this into a top-level post for a while, but I’ll just dump a rough summary in a comment here quickly for now.)
FLOPS are a macro-level performance characteristic of a system. If I claim that a system is capable of 1 million FLOPS, that means something very precise in computer engineering terms. It means that the system can take 1 million pairs of floating point numbers (at some precision) and return 1 million results of multiplying or adding those pairs, each and every second.
OTOH, if I claim that a system is performing N million “synaptic ops” per second, there’s not a clear high-level / outward-facing / measurable performance characteristic that translates to. For large enough N, suitably arranged, you get human-level general cognition, which of course can be used to perform all sorts of behaviors that can be easily quantified into crisply specifiable performance characteristics. But that doesn’t mean that the synaptic ops themselves can be treated as a macro-level performance characteristic the way FLOPS can, even if each and every one of them really is maximally efficient on an individual basis and strictly necessary for the task at hand.
When you try to compare the brain and silicon-based systems in strictly valid but naive ways, you get results that are actually valid but not particularly useful or informative:
On the literal task of multiplying floats, almost any silicon system is vastly more efficient than a human brain, by any metric. Silicon can be used to multiply millions of floats per second for relatively tiny amounts of energy, whereas if you give a human a couple of floats and ask them to multiply them together, it would probably take several minutes, assuming they get the answer right and don’t wander off out of boredom, for a rate of <0.008 FLOPS.
On higher-level tasks like image recognition, visual processing, board games, etc. humans and silicon-based systems are more competitive, depending on the difficulty of the task and how the results are measured (energy, wall clock time, correctness, ELO etc.)
For sufficiently cognitively difficult tasks which can’t currently be performed in silico at all, there’s no rigorous way to do the comparison naively, except by speculating about how much computation future systems or algorithms might use. The time / energy / synaptic ops budget needed to e.g. solve a novel olympiad-level math problem for a human is roughly known and calculable, the time / energy / FLOPS budget for a silicon system to do the same is unknown (but in any case, it probably depends heavily on how you use the FLOPS, in terms of arranging them to perform high-level cognition). We can speculate about how efficient possible arrangements might be compared to brain, but for the actual arrangements that people today know how to write code for even in theory (e.g. AIXI), the efficiency ratio is essentially infinite (AIXI is incomputable, human brains are almost certainly not doing anything incomputable.)
I don’t think Jacob’s analysis is totally wrong or useless, but one must be very careful to keep track of what it is that is being compared, and why. Joe Carlsmith does this well in How Much Computational Power Does It Take to Match the Human Brain? (which Jacob cites). I think much of Jacob’s analysis is an attempt at what Joe calls the mechanistic method:
Estimate the FLOP/s required to model the brain’s mechanisms at a level of detail adequate to replicate task-performance (the“mechanistic method”).
(Emphasis mine.) Joe is very careful to be clear that the analysis is about modeling the brain’s mechanisms (i.e. simulation), rather than attempting to directly or indirectly compare the “amount of computation” performed by brains or CPUs, or their relative efficiency at performing this purported / estimated / equated amount of computation.
Other methods, which Jacob also uses, (e.g. the limit method) in Joe’s analysis can be used to upper bound the FLOPS required for human-level general cognition, but Joe correctly points out that this analysis can’t be used directly to place a lower bound on how efficiently a difficult-to-crisply-specify task (e.g. high-level general cognition) can be performed:
None of these methods are direct guides to the minimum possible FLOP/s budget, as the most efficient ways of performing tasks need not resemble the brain’s ways, or those of current artificial systems. But if sound, these methods would provide evidence that certain budgets are, at least, big enough (if you had the right software, which may be very hard to create – see discussion in section 1.3).2
I might expand on this more in the future, but I have procrastinated on it so far mainly because I don’t actually think the question these kinds of analyses attempt to answer is that relevant to important questions about AGI capabilities or limits: in my view, the minimal hardware required for creating superhuman AGI very likely already exists, probably many times over.
My ownview is that you only need something a little bit smarter than the smartest humans, in some absolute sense, in order to re-arrange most of the matter and energy in the universe (almost) arbitrarily. If you can build epsilon-smarter-than-human-level AGI using (say) an energy budget of 1,000 W at runtime, you or the AGI itself can probably then figure out how to scale to 10,000 W or 100,000 W relatively easily (relative to the task of creating the 1,000 W AGI in the first place). And my guess is that the 100,000 W system is just sufficient for almost anything you or it wants to do, at least in the absence of other, adversarial agents of similar intelligence.
Rigorous, gears-level analyses can provide more and more precise lower bounds on the exact hardware and energy requirements for general intelligence, but these bounds are generally non-constructive. To actually advance capabilities (or make progress on alignment) my guess is that you need to do “great original natural philosophy”, as Tsvi calls it. If you do enough philosophy carefully and precisely before you (or anyone else) builds a 100,000 W AGI, you get a glorious transhuman future, if not, you probably get squiggles. And I think analyses like Joe’s and Jacob’s show that the hardware and energy required to build a merely human-level AGI probably already exists, even if it comes with a few OOM (or more) energy efficiency penalty relative to the brain. As manycommenters on Jacob’s original post pointed out, silicon-based systems are already capable of making productive use of vastly more readily-available energy than a biological brain.
I think it does matter how efficient AI can get in using energy to fuel computation, in that I think it provides an important answer to the question over whether AI will be distributed widely, and whether we can realistically control AI distribution and creation.
If it turns out that creating superhuman AI is possible without much use of energy by individuals in their basement, then long term, controlling AI becomes essentially impossible, and we will have to confront a world where the government isn’t going to reliably control AI by default. Essentially, Eliezer’s initial ideas about the ability to create very strong technology in your basement may eventually become reality, just with a time delay.
If it turns out that any AI must use a minimum of say 10,000 watts or more, then there is hope for controlling AI creation and distribution long term.
And this matters both in scenarios where existential risk mostly comes from individuals, and scenarios where existential risk doesn’t matter, but what will happen in a world where superhuman AI is created.
If it turns out that any AI must use a minimum of say 10,000 watts or more, then there is hope for controlling AI creation and distribution long term.
Note, 1 kW (50-100x human brain wattage) is roughly the power consumption of a very beefy desktop PC, and 10 kW is roughly the power consumption of a single rack in a datacenter. Even ~megawatt scale AI (100 racks) could fit pretty easily within many existing datacenters, or within a single entity’s mid-size industrial-scale basement, at only moderate cost.
Yeah, this isn’t enough to stop companies from producing useful AI, but it does mostly mean we can hope to avoid scenarios where single individuals can reliably build AI, meaning that controlling AI in scenarios where individuals, but not companies are the problem for existential risk is possible. It’s also relevant for other questions not focused on existential risk as well.
My own view is that you only need something a little bit smarter than the smartest humans, in some absolute sense, in order to re-arrange most of the matter and energy in the universe (almost) arbitrarily
Does this imply that a weakly superhuman AGI can solve alignment?
Thanks for facilitating this! I found Steven’s post and spxtr’s comments particularly insightful.
I still think there’s a more fundamental issue with Jacob’s analysis, which is also present in some other works that characterize the brain as a substrate for various kinds of computation usually performed in silicon.
Namely, Jacob (and others) are implicitly or explicitly comparing FLOPS with “synaptic ops”, but these quantities are fundamentally incomparable. (I’ve been meaning to turn this into a top-level post for a while, but I’ll just dump a rough summary in a comment here quickly for now.)
FLOPS are a macro-level performance characteristic of a system. If I claim that a system is capable of 1 million FLOPS, that means something very precise in computer engineering terms. It means that the system can take 1 million pairs of floating point numbers (at some precision) and return 1 million results of multiplying or adding those pairs, each and every second.
OTOH, if I claim that a system is performing N million “synaptic ops” per second, there’s not a clear high-level / outward-facing / measurable performance characteristic that translates to. For large enough N, suitably arranged, you get human-level general cognition, which of course can be used to perform all sorts of behaviors that can be easily quantified into crisply specifiable performance characteristics. But that doesn’t mean that the synaptic ops themselves can be treated as a macro-level performance characteristic the way FLOPS can, even if each and every one of them really is maximally efficient on an individual basis and strictly necessary for the task at hand.
When you try to compare the brain and silicon-based systems in strictly valid but naive ways, you get results that are actually valid but not particularly useful or informative:
On the literal task of multiplying floats, almost any silicon system is vastly more efficient than a human brain, by any metric. Silicon can be used to multiply millions of floats per second for relatively tiny amounts of energy, whereas if you give a human a couple of floats and ask them to multiply them together, it would probably take several minutes, assuming they get the answer right and don’t wander off out of boredom, for a rate of <0.008 FLOPS.
On higher-level tasks like image recognition, visual processing, board games, etc. humans and silicon-based systems are more competitive, depending on the difficulty of the task and how the results are measured (energy, wall clock time, correctness, ELO etc.)
For sufficiently cognitively difficult tasks which can’t currently be performed in silico at all, there’s no rigorous way to do the comparison naively, except by speculating about how much computation future systems or algorithms might use. The time / energy / synaptic ops budget needed to e.g. solve a novel olympiad-level math problem for a human is roughly known and calculable, the time / energy / FLOPS budget for a silicon system to do the same is unknown (but in any case, it probably depends heavily on how you use the FLOPS, in terms of arranging them to perform high-level cognition). We can speculate about how efficient possible arrangements might be compared to brain, but for the actual arrangements that people today know how to write code for even in theory (e.g. AIXI), the efficiency ratio is essentially infinite (AIXI is incomputable, human brains are almost certainly not doing anything incomputable.)
I don’t think Jacob’s analysis is totally wrong or useless, but one must be very careful to keep track of what it is that is being compared, and why. Joe Carlsmith does this well in How Much Computational Power Does It Take to Match the Human Brain? (which Jacob cites). I think much of Jacob’s analysis is an attempt at what Joe calls the mechanistic method:
(Emphasis mine.) Joe is very careful to be clear that the analysis is about modeling the brain’s mechanisms (i.e. simulation), rather than attempting to directly or indirectly compare the “amount of computation” performed by brains or CPUs, or their relative efficiency at performing this purported / estimated / equated amount of computation.
Other methods, which Jacob also uses, (e.g. the limit method) in Joe’s analysis can be used to upper bound the FLOPS required for human-level general cognition, but Joe correctly points out that this analysis can’t be used directly to place a lower bound on how efficiently a difficult-to-crisply-specify task (e.g. high-level general cognition) can be performed:
I might expand on this more in the future, but I have procrastinated on it so far mainly because I don’t actually think the question these kinds of analyses attempt to answer is that relevant to important questions about AGI capabilities or limits: in my view, the minimal hardware required for creating superhuman AGI very likely already exists, probably many times over.
My own view is that you only need something a little bit smarter than the smartest humans, in some absolute sense, in order to re-arrange most of the matter and energy in the universe (almost) arbitrarily. If you can build epsilon-smarter-than-human-level AGI using (say) an energy budget of 1,000 W at runtime, you or the AGI itself can probably then figure out how to scale to 10,000 W or 100,000 W relatively easily (relative to the task of creating the 1,000 W AGI in the first place). And my guess is that the 100,000 W system is just sufficient for almost anything you or it wants to do, at least in the absence of other, adversarial agents of similar intelligence.
Rigorous, gears-level analyses can provide more and more precise lower bounds on the exact hardware and energy requirements for general intelligence, but these bounds are generally non-constructive. To actually advance capabilities (or make progress on alignment) my guess is that you need to do “great original natural philosophy”, as Tsvi calls it. If you do enough philosophy carefully and precisely before you (or anyone else) builds a 100,000 W AGI, you get a glorious transhuman future, if not, you probably get squiggles. And I think analyses like Joe’s and Jacob’s show that the hardware and energy required to build a merely human-level AGI probably already exists, even if it comes with a few OOM (or more) energy efficiency penalty relative to the brain. As many commenters on Jacob’s original post pointed out, silicon-based systems are already capable of making productive use of vastly more readily-available energy than a biological brain.
I think it does matter how efficient AI can get in using energy to fuel computation, in that I think it provides an important answer to the question over whether AI will be distributed widely, and whether we can realistically control AI distribution and creation.
If it turns out that creating superhuman AI is possible without much use of energy by individuals in their basement, then long term, controlling AI becomes essentially impossible, and we will have to confront a world where the government isn’t going to reliably control AI by default. Essentially, Eliezer’s initial ideas about the ability to create very strong technology in your basement may eventually become reality, just with a time delay.
If it turns out that any AI must use a minimum of say 10,000 watts or more, then there is hope for controlling AI creation and distribution long term.
And this matters both in scenarios where existential risk mostly comes from individuals, and scenarios where existential risk doesn’t matter, but what will happen in a world where superhuman AI is created.
Note, 1 kW (50-100x human brain wattage) is roughly the power consumption of a very beefy desktop PC, and 10 kW is roughly the power consumption of a single rack in a datacenter. Even ~megawatt scale AI (100 racks) could fit pretty easily within many existing datacenters, or within a single entity’s mid-size industrial-scale basement, at only moderate cost.
Yeah, this isn’t enough to stop companies from producing useful AI, but it does mostly mean we can hope to avoid scenarios where single individuals can reliably build AI, meaning that controlling AI in scenarios where individuals, but not companies are the problem for existential risk is possible. It’s also relevant for other questions not focused on existential risk as well.
Does this imply that a weakly superhuman AGI can solve alignment?
Thank you Max, you make some very good points.