A mouse brain is approximately 3 orders of magnitude smaller than a human brain. The neurons that make it up are quite similar, with the exception of a few changes such as human cortical neurons tending to have more synapses.
In his paper, ″, Penrose discusses the hypothesis that the pattern of protein subunits in microtubules allows for additional processing in the brain. His estimate of per neuron compute is 1e16 FLOPs, for a total human brain compute of 1e27 FLOPs.
This would mean that a mouse brain has about 1e24 FLOPs of compute.
Fruit Fly
In 2023, using the data from 2017 (above), the full brain connectome (for a female) was made available, containing roughly 5x10^7 chemical synapses between ~130,000 neurons. - https://en.wikipedia.org/wiki/Drosophila_connectome
A more extreme case is that of the fruit fly. A fruit fly has around 1e5 neurons. With Roger Penrose’s 10^16 FLOP/s/neuron compute estimate, this works out to a fruit fly brain having around 1.3e11 FLOP/s.
GPUs of 2023
A back-of-the-envelope calculation based on market size, price-performance, hardware lifespan estimates, and the sizes of Google’s data centers estimates that there is around 3.98 * 1e21 FLOP/s of computing capacity on GPUs and TPUs as of Q1 2023.
A single inference instance of LLama 405B can be run on an 8 H100 GPU server. So, about 1e15 FLOP/s * 8 = 8e15 FLOP/s
Nathan’s Estimate
My upper estimate of the compute of the human brain is about 1e16 FLOPs. My ‘best guess’ Fermi estimate is about 1e15. In other words, my best Fermi estimate is that the human brain is doing about the same amount of computation as an H100 at max throughput. My upper estimate is about 10x that, so 10 H100 GPUs. If I weren’t restricting myself to nearest order of magnitude, I’d guess somewhere in between 1e15 and 1e16.
Training
So for a human brain to do the compute that was used to train Llama 3.1 405B, it would require 3e6 − 3e7 hours of waking thought. So somewhere between 500 years and 5000 years.
A mouse brain therefore, would take around 5e5 to 5e6 years.
Inference
My estimate is that a human brain should be able to run a single LLama 405B inference instance somewhere between 1/10th speed or about normal speed. Even under my worst case assumptions, a human brain should be more than sufficient to run a LLama 70B instance.
A mouse brain would need to run LLama 405B at 1/1e3 to 1/1e4 speed (and I’d be concerned about insufficient RAM equivalent to even hold enough of it to enable this! That’s a whole different discussion).
Roger’s Estimate
Training
By Roger Penrose’s estimate, a mouse brain should be able to train an LLM equivalent to LLama 405B in 10 seconds. A human brain should be able to train 100 such models every second.
Inference
Roger’s estimate suggests a single mouse brain should be able to run over 100 million (~1.2e8) copies of LLama 405B simultaneously at full speed.
A human brain could run 1000x that many, so about 1e11 copies simultaneously.
You’d need to network 1e4 fruit fly brains together to run a single copy of LLama 405B. Or a mere 200 fruit flies to run Llama 8B. What a bargain!
Baboons
From an ethological perspective, we can analyze the behaviors of animals in the wild in respect to computationally difficult problems. For example, baboons solving the traveling salesman problem of choosing which resource areas to visit over time.
If the baboons had 10^26 flops (ungenerously estimating them at 1/10th of humans) of compute at their disposal, it would be trivial for them to calculate a better solution to their resource acquisition problem. Why don’t they?
Empirical resolution
So, there is limited flexibility of compute in biological brains. Most of the neurons in the human brain are wired up in relatively strict motifs to accomplish certain particular types of computation. So you can’t actually test any of these theories as stated. The exception to this general rule is that the cortex of infant mammals (and birds, etc) is relatively flexible, and about as close to pure ‘general compute’ that biological brains get. So if you wired up an infant mouse with a Brain-Computer-Interface connecting its cortex to a computer, and removed the other sensory inputs and outputs, it’s theoretically possible to get some estimates. It would also be necessary to the compute estimates to be just for the cortex instead of the whole brain.
If Roger Penrose’s ideas were right, then a single long-lived rodent (e.g. naked mole rat) could be worth hundreds of billions of dollars of compute. Seems like if the scientific community suspected this were true, then at least a few scientists would be trying to develop such a BCI system?
Penrose may be a coauthor, but the estimate would really be due to his colleague Stuart Hameroff (who I worked for once), an anesthesiologist who has championed the idea of microtubules as a locus of cognition and consciousness.
As I said in the other thread, even if microtubules do have a quantum computational role, my own expectation is that they would each only contribute a few “logical qubits” at best, e.g. via phase factors created by quasiparticles propagating around the microtubule cylinder, something which might be topologically protected against room-temperature thermal fluctuations.
But there are many ideas about their dynamics, so, I am not dogmatic about this scenario.
Seems like if the scientific community suspected this were true, then at least a few scientists would be trying to develop such a BCI system?
There’s no mystery as to whether the scientific community suspects that the Hameroff-Penrose theory is correct. It is a well-known hypothesis in consciousness studies, but it wouldn’t have too many avowed adherents beyond its inventors: perhaps some people whose own research program overlaps with it, and some other scattered sympathizers.
It could be compared to the idea that memories are stored in RNA, another hypothesis that has been around for decades, and which has a pop-culture charisma far beyond its scientific acceptance.
So it’s not a mainstream hypothesis, but it is known enough, and overlaps with a broader world of people working on microtubule physics, biocomputation, quantum biology, and other topics. See what Google Scholar returns for “microtubule biocomputer” and scroll through a few pages. You will find, for example, a group in Germany trying to create artificial microtubule lattices for “network-based biocomputation” (which is about swarms of kinesins walking around the lattice, like mice in a maze looking for the exit), a book on using actin filaments (a relative of the microtubule) for “revolutionary computing systems”, and many other varied proposals.
I don’t see anyone specifically trying to hack mouse neurons in order to make novel microtubular deep-learning systems, but that just means you’re going to be the godfather of that particular concept (like Petr Vopěnka, whose“principle”started as a joke he didn’t believe).
[note that this is not what Mitchell_Porter and I are disagreeing over in this related comment thread: https://www.lesswrong.com/posts/uPi2YppTEnzKG3nXD/nathan-helm-burger-s-shortform?commentId=AKEmBeXXnDdmp7zD6 ]
Contra Roger Penrose on estimates of brain compute
[numeric convention used here is that <number1>e<number2> means number1 * 10 ^ number2. Examples: 1e2 = 100, 2e3 = 2000, 5.3e2 = 530, 5.3e-2 = 0.053]
Mouse
The cerebral cortex of a mouse has around 8–14 million neurons while in those humans there are more than 10–15 billion—https://en.wikipedia.org/wiki/Mouse_brain
A mouse brain is approximately 3 orders of magnitude smaller than a human brain. The neurons that make it up are quite similar, with the exception of a few changes such as human cortical neurons tending to have more synapses.
In his paper, ″, Penrose discusses the hypothesis that the pattern of protein subunits in microtubules allows for additional processing in the brain. His estimate of per neuron compute is 1e16 FLOPs, for a total human brain compute of 1e27 FLOPs.
This would mean that a mouse brain has about 1e24 FLOPs of compute.
Fruit Fly
In 2023, using the data from 2017 (above), the full brain connectome (for a female) was made available, containing roughly 5x10^7 chemical synapses between ~130,000 neurons. - https://en.wikipedia.org/wiki/Drosophila_connectome
https://www.biorxiv.org/content/10.1101/2023.06.27.546656v2
1.3e5 neurons * 1e16 FLOPs/neuron = 1.3e11 FLOPs
A more extreme case is that of the fruit fly. A fruit fly has around 1e5 neurons. With Roger Penrose’s 10^16 FLOP/s/neuron compute estimate, this works out to a fruit fly brain having around 1.3e11 FLOP/s.
GPUs of 2023
A back-of-the-envelope calculation based on market size, price-performance, hardware lifespan estimates, and the sizes of Google’s data centers estimates that there is around 3.98 * 1e21 FLOP/s of computing capacity on GPUs and TPUs as of Q1 2023.
source: https://wiki.aiimpacts.org/doku.php?id=ai_timelines:hardware_and_ai_timelines:computing_capacity_of_all_gpus_and_tpus
So, Roger Penrose’s estimate claims that a single mouse brain has 1000 times more computing power of all the GPUs on Earth in Q1 2023.
Training a Large Language Model
″… H100’s TF32 precision to achieve one petaflop of throughput …”
https://www.nvidia.com/en-us/data-center/h100/
A petaflop is a unit of computing speed equal to one thousand million million (1015)floating-point operations per second.
1e15 * 60 seconds/min * 60 min/hour = 3.6e17 FLOP/s per H100 GPU hour.
Training Llama 3.1 405B utilized 3.08e7 GPU hours of computation on H100-80GB GPUs. So about 1e25 FLOP/s.
https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md
A single inference instance of LLama 405B can be run on an 8 H100 GPU server. So, about 1e15 FLOP/s * 8 = 8e15 FLOP/s
Nathan’s Estimate
My upper estimate of the compute of the human brain is about 1e16 FLOPs. My ‘best guess’ Fermi estimate is about 1e15. In other words, my best Fermi estimate is that the human brain is doing about the same amount of computation as an H100 at max throughput. My upper estimate is about 10x that, so 10 H100 GPUs. If I weren’t restricting myself to nearest order of magnitude, I’d guess somewhere in between 1e15 and 1e16.
Training
So for a human brain to do the compute that was used to train Llama 3.1 405B, it would require 3e6 − 3e7 hours of waking thought. So somewhere between 500 years and 5000 years.
A mouse brain therefore, would take around 5e5 to 5e6 years.
Inference
My estimate is that a human brain should be able to run a single LLama 405B inference instance somewhere between 1/10th speed or about normal speed. Even under my worst case assumptions, a human brain should be more than sufficient to run a LLama 70B instance.
A mouse brain would need to run LLama 405B at 1/1e3 to 1/1e4 speed (and I’d be concerned about insufficient RAM equivalent to even hold enough of it to enable this! That’s a whole different discussion).
Roger’s Estimate
Training
By Roger Penrose’s estimate, a mouse brain should be able to train an LLM equivalent to LLama 405B in 10 seconds. A human brain should be able to train 100 such models every second.
Inference
Roger’s estimate suggests a single mouse brain should be able to run over 100 million (~1.2e8) copies of LLama 405B simultaneously at full speed.
A human brain could run 1000x that many, so about 1e11 copies simultaneously.
You’d need to network 1e4 fruit fly brains together to run a single copy of LLama 405B. Or a mere 200 fruit flies to run Llama 8B. What a bargain!
Baboons
From an ethological perspective, we can analyze the behaviors of animals in the wild in respect to computationally difficult problems. For example, baboons solving the traveling salesman problem of choosing which resource areas to visit over time.
https://www.youtube.com/watch?v=vp7EbO-IXtg&t=198s
If the baboons had 10^26 flops (ungenerously estimating them at 1/10th of humans) of compute at their disposal, it would be trivial for them to calculate a better solution to their resource acquisition problem. Why don’t they?
Empirical resolution
So, there is limited flexibility of compute in biological brains. Most of the neurons in the human brain are wired up in relatively strict motifs to accomplish certain particular types of computation. So you can’t actually test any of these theories as stated. The exception to this general rule is that the cortex of infant mammals (and birds, etc) is relatively flexible, and about as close to pure ‘general compute’ that biological brains get. So if you wired up an infant mouse with a Brain-Computer-Interface connecting its cortex to a computer, and removed the other sensory inputs and outputs, it’s theoretically possible to get some estimates. It would also be necessary to the compute estimates to be just for the cortex instead of the whole brain.
If Roger Penrose’s ideas were right, then a single long-lived rodent (e.g. naked mole rat) could be worth hundreds of billions of dollars of compute. Seems like if the scientific community suspected this were true, then at least a few scientists would be trying to develop such a BCI system?
A few comments:
The name of the paper is currently missing…
Penrose may be a coauthor, but the estimate would really be due to his colleague Stuart Hameroff (who I worked for once), an anesthesiologist who has championed the idea of microtubules as a locus of cognition and consciousness.
As I said in the other thread, even if microtubules do have a quantum computational role, my own expectation is that they would each only contribute a few “logical qubits” at best, e.g. via phase factors created by quasiparticles propagating around the microtubule cylinder, something which might be topologically protected against room-temperature thermal fluctuations.
But there are many ideas about their dynamics, so, I am not dogmatic about this scenario.
There’s no mystery as to whether the scientific community suspects that the Hameroff-Penrose theory is correct. It is a well-known hypothesis in consciousness studies, but it wouldn’t have too many avowed adherents beyond its inventors: perhaps some people whose own research program overlaps with it, and some other scattered sympathizers.
It could be compared to the idea that memories are stored in RNA, another hypothesis that has been around for decades, and which has a pop-culture charisma far beyond its scientific acceptance.
So it’s not a mainstream hypothesis, but it is known enough, and overlaps with a broader world of people working on microtubule physics, biocomputation, quantum biology, and other topics. See what Google Scholar returns for “microtubule biocomputer” and scroll through a few pages. You will find, for example, a group in Germany trying to create artificial microtubule lattices for “network-based biocomputation” (which is about swarms of kinesins walking around the lattice, like mice in a maze looking for the exit), a book on using actin filaments (a relative of the microtubule) for “revolutionary computing systems”, and many other varied proposals.
I don’t see anyone specifically trying to hack mouse neurons in order to make novel microtubular deep-learning systems, but that just means you’re going to be the godfather of that particular concept (like Petr Vopěnka, whose “principle” started as a joke he didn’t believe).
That would indeed be hilarious. The Helm-Burger-Porter microtubule computer would haunt me the rest of my days!