And “analog multiplication down to two decimal places” is the operation that is purportedly being carried out almost as efficiently as physically possible by
I am not certain it is being carried “almost as efficiently as physically possible”, assuming you mean thermodynamic efficiency (even accepting you meant thermodynamic efficiency only for irreversible computation), my belief is more that the brain and its synaptic elements are reasonably efficient in a pareto tradeoff sense.
But any discussion around efficiency must make some starting assumptions about what computations the system may be performing. We now have a reasonable amount of direct and indirect evidence—direct evidence from neuroscience, indirect evidence from DL—that allows us some confidence that the brain is conventional (irreversible, non quantum), and is basically very similar to an advanced low power DL accelerator built out of nanotech replicators. (and the clear obvious trend in hardware design is towards the brain)
So starting with that frame ..
Having any physical equivalent of an analog multiplication fundamentally requires 100,000 times the thermodynamic energy to erase 1 bit?
A synaptic op is the equivalent of reading an 8b-ish weight from memory, ‘multiplying’ by the incoming spike value, propagating the output down the wire, updating neurotransmitter receptors (which store not just the equivalent of the weight, but the bayesian distribution params on the weight, equivalent to gradient momentum etc), back-propagating spike (in some scenarios), spike decoding (for nonlinear spike timing codes), etc.
It just actually does a fair amount of work, and if you actually query the research literature to see how many transistors that would take it is something like 10k to 100k or more, each of which minimally uses 1eV per op * 10 for interconnect, according to the best micromodels of circuit limits (cavin/Zhirnov).
The analog multiplier and gear is very efficient (especially in space) for low SNRs, but it scales poorly (exponentially) with bit precision (equivalent SNR). From the last papers I recall 8b is the crossover point where digital wins in energy and perhaps size. Below that analog dominates. There are numerous startups working on analog hardware to replace GPUs for low bit precision multipliers, chasing the brain, but its extremely difficult and IMHO may. not be worth it without nanotech.
in order to transmit the impulse at 1m/s?
The brain only runs at 100hz and the axon conduction velocity is optimized just so that every brain region can connect to distal regions without significant delay (delay on order of a millisecond or so).
So the real question is then just why 100hz—which I also answer in brain efficiency. If you have a budget of 10W you can spend that running a very small NN very fast or a very large NN at lower speeds, and the latter seems more useful for biology. Digital minds obviously can spend the energy cost to run at fanastic speeds—and GPT4 was only possible because its NN can run vaguely ~10000x faster than the brain (for training).
I’ll end with an interesting quote from Hinton[1]:
The separation of software from hardware is one of the foundations of Computer Science and it
has many benefits. It makes it possible to study the properties of programs without worrying about
electrical engineering. It makes it possible to write a program once and copy it to millions of
computers. If, however, we are willing to abandon immortality it should be possible to achieve huge
savings in the energy required to perform a computation and in the cost of fabricating the hardware
that executes the computation. We can allow large and unknown variations in the connectivity and
non-linearities of different instances of hardware that are intended to perform the same task and
rely on a learning procedure to discover parameter values that make effective use of the unknown
properties of each particular instance of the hardware. These parameter values are only useful for that
specific hardware instance, so the computation they perform is mortal: it dies with the hardware.
I am not certain it is being carried “almost as efficiently as physically possible”, assuming you mean thermodynamic efficiency (even accepting you meant thermodynamic efficiency only for irreversible computation), my belief is more that the brain and its synaptic elements are reasonably efficient in a pareto tradeoff sense.
But any discussion around efficiency must make some starting assumptions about what computations the system may be performing. We now have a reasonable amount of direct and indirect evidence—direct evidence from neuroscience, indirect evidence from DL—that allows us some confidence that the brain is conventional (irreversible, non quantum), and is basically very similar to an advanced low power DL accelerator built out of nanotech replicators. (and the clear obvious trend in hardware design is towards the brain)
So starting with that frame ..
A synaptic op is the equivalent of reading an 8b-ish weight from memory, ‘multiplying’ by the incoming spike value, propagating the output down the wire, updating neurotransmitter receptors (which store not just the equivalent of the weight, but the bayesian distribution params on the weight, equivalent to gradient momentum etc), back-propagating spike (in some scenarios), spike decoding (for nonlinear spike timing codes), etc.
It just actually does a fair amount of work, and if you actually query the research literature to see how many transistors that would take it is something like 10k to 100k or more, each of which minimally uses 1eV per op * 10 for interconnect, according to the best micromodels of circuit limits (cavin/Zhirnov).
The analog multiplier and gear is very efficient (especially in space) for low SNRs, but it scales poorly (exponentially) with bit precision (equivalent SNR). From the last papers I recall 8b is the crossover point where digital wins in energy and perhaps size. Below that analog dominates. There are numerous startups working on analog hardware to replace GPUs for low bit precision multipliers, chasing the brain, but its extremely difficult and IMHO may. not be worth it without nanotech.
The brain only runs at 100hz and the axon conduction velocity is optimized just so that every brain region can connect to distal regions without significant delay (delay on order of a millisecond or so).
So the real question is then just why 100hz—which I also answer in brain efficiency. If you have a budget of 10W you can spend that running a very small NN very fast or a very large NN at lower speeds, and the latter seems more useful for biology. Digital minds obviously can spend the energy cost to run at fanastic speeds—and GPT4 was only possible because its NN can run vaguely ~10000x faster than the brain (for training).
I’ll end with an interesting quote from Hinton[1]:
The Forward-Forward Algorithm -section 8