I mean that deep learning and the brain operate off of the same principles: function approximation via high dimensional geometric inference of circuits. They are in the same part of program-space: feed forward connections through a neuron, and feed back connections (from somewhere) to train it; cost functions that train neurons into circuits which satisfy an objective; on the higher level, multiple layers, gating, recurrence, etc architectural features like local connectivity, fast-weight memory, and internal attention. People in machine learning were surprised to find that backprop is enough to train neural networks; it was widely expected that it wouldn’t work, but it’s been discovered that neural networks provide various properties that provide good geometric priors about the world we live in—an intuition pump is “if you have enough dimensions, you always have some way to do slightly better”, ie there are nearly no local minima. This isn’t actually fully true in general, but neural networks seem to be structures that make it true, and I think that this basic principle is also what makes the brain work.
So it looks like “connectome” didn’t mean the thing I remembered it meaning—I thought it meant “wiring patterns”, but it in fact means “all wiring in the entire thing”. I agree that, in this corrected sense, the connectome is almost certainly dynamic. And deep learning certainly covers that dynamism, it’s just the fact of weights being learned, and being allowed to be 0. The brain probably has some form of regularization force pushing connections to disappear—pushing weights to 0 - if they aren’t useful enough to keep the wires in place. But that’s a common effect of regularization in deep learning too; L2 regularization does this to some degree, and L1 regularization does it harder.
Deep learning doesn’t learn as many weights. Neurons seem to be willing to form 10^3 to 10^4 (typical in the cortex) up to 10^5 connections (in the cerebellum). The equivalent neuron in deep learning would have to have far more weights, because in deep learning, zero weights still exist in the matrix. The brain is made out of circuits that can actually get rid of zeros.
Do you disagree with my points such that you think it’s worth exploring where our disagreement is?
I mean that deep learning and the brain operate off of the same principles: function approximation via high dimensional geometric inference of circuits. They are in the same part of program-space: feed forward connections through a neuron, and feed back connections (from somewhere) to train it; cost functions that train neurons into circuits which satisfy an objective; on the higher level, multiple layers, gating, recurrence, etc architectural features like local connectivity, fast-weight memory, and internal attention. People in machine learning were surprised to find that backprop is enough to train neural networks; it was widely expected that it wouldn’t work, but it’s been discovered that neural networks provide various properties that provide good geometric priors about the world we live in—an intuition pump is “if you have enough dimensions, you always have some way to do slightly better”, ie there are nearly no local minima. This isn’t actually fully true in general, but neural networks seem to be structures that make it true, and I think that this basic principle is also what makes the brain work.
So it looks like “connectome” didn’t mean the thing I remembered it meaning—I thought it meant “wiring patterns”, but it in fact means “all wiring in the entire thing”. I agree that, in this corrected sense, the connectome is almost certainly dynamic. And deep learning certainly covers that dynamism, it’s just the fact of weights being learned, and being allowed to be 0. The brain probably has some form of regularization force pushing connections to disappear—pushing weights to 0 - if they aren’t useful enough to keep the wires in place. But that’s a common effect of regularization in deep learning too; L2 regularization does this to some degree, and L1 regularization does it harder.
Deep learning doesn’t learn as many weights. Neurons seem to be willing to form 10^3 to 10^4 (typical in the cortex) up to 10^5 connections (in the cerebellum). The equivalent neuron in deep learning would have to have far more weights, because in deep learning, zero weights still exist in the matrix. The brain is made out of circuits that can actually get rid of zeros.
Do you disagree with my points such that you think it’s worth exploring where our disagreement is?
I agree that there is a tonne of that going on in the brain. I think I doubt it is all that is there.
Neuroplasiticity in my mind is not just about modifying weights. New neurons are being added constantly. I don’t think deep learning covers that.