Here are clarifications for a couple minor things I was confused about while reading:
a GPU is way below the power of the human brain. You need something like 100,000 or a million to match it, so we are off by a huge factor here.
I was trying to figure out where LeCun’s 100,000+ claim is coming from, and I found this 2017 article which is paywalled but the subheading implies that he’s focusing on the 10^14 synapses in a human brain, and comparing that to the number of neuron-to-neuron connections that a GPU can handle.
(If so, I strongly disagree with that comparison, but I don’t want to get into that in this little comment.)
Francois Chollet says “The actual information input of the visual system is under 1MB/s”.
I don’t think Chollet is directly responding to LeCun’s point, because Chollet is saying that optical information is compressible to <1MB/s, but LeCun is comparing uncompressed human optical bits to uncompressed LLM text bits. And the text bits are presumably compressible too!
It’s possible that the compressibility (useful information content per bit) of optical information is very different from the compressibility of text information, in which case LeCun’s comparison is misleading, but neither LeCun nor Chollet is making claims one way or the other about that, AFAICT.
Here are clarifications for a couple minor things I was confused about while reading:
I was trying to figure out where LeCun’s 100,000+ claim is coming from, and I found this 2017 article which is paywalled but the subheading implies that he’s focusing on the 10^14 synapses in a human brain, and comparing that to the number of neuron-to-neuron connections that a GPU can handle.
(If so, I strongly disagree with that comparison, but I don’t want to get into that in this little comment.)
I don’t think Chollet is directly responding to LeCun’s point, because Chollet is saying that optical information is compressible to <1MB/s, but LeCun is comparing uncompressed human optical bits to uncompressed LLM text bits. And the text bits are presumably compressible too!
It’s possible that the compressibility (useful information content per bit) of optical information is very different from the compressibility of text information, in which case LeCun’s comparison is misleading, but neither LeCun nor Chollet is making claims one way or the other about that, AFAICT.
Thanks! I added a note about LeCun’s 100,000 claim and just dropped the Chollet reference since it was misleading.