This piece is super interesting, especially the toy models.
A few clarifying questions:
‘And not just any learning algorithm! The neocortex runs a learning algorithm, but it’s an algorithm that picks outputs to maximize reward, not outputs to anticipate matched supervisory signals. This needs to be its own separate brain module, I think.’
-- Why does it need to be its own separate module? Can you expand on this? And even if separate modules are useful (as per your toy models and different inputs, couldn’t the neocortex also be running lookup table like auto or hetero-associative learning).
“Parallel fibers carry the context signals, Purkinje cells produce the output signals, and climbing fibers carry the shoulda signals, and the synapse strengths between a parallel fiber and a Purkinje cell is modified as a function of how recently the parallel fiber and climbing fiber have fired.”
Can you explain how the accelerator works in more detail (esp as you use it in the later body and cognition toy models 5 and 6)? Why is the cerebellum faster at producing outputs than the neocortex? How does the neocortex carry the “shoulda” signal? Finally I’m confused by this line:
“You can’t just take any incoming and outgoing signal lines and pair them up! …Well, in an accelerator, you can! Let’s say the output line in the diagram controls Muscle #847. Which shoulda line needs to be paired up with that? The answer is: it doesn’t matter! Just take any neocortex output signal and connect it. And then the neocortex will learn over time that that output line is effectively a controller for Muscle #847.”
-- This suggests that the neocortex can learn the cerebellar mapping and short-circuit to use it? Why does it need to go through the cerebellum to do this? Rather than via the motor cortex and efferent connections back to the muscles?
Maybe you’re ahead of me, but it took me until long after this post—just a couple weeks ago—to realize that you can take a neural circuit set up for RL, and jury-rig it to do supervised learning instead.
I think this is a big part of the story behind what the vmPFC is doing. And, in a certain sense, the amygdala too. More on this in a forthcoming post.
couldn’t the neocortex also be running lookup table like auto or hetero-associative learning … Why is the cerebellum faster at producing outputs than the neocortex?
I think of the neocortex as doing analysis-by-synthesis—it searches through a space of generative models for one that matches the input data. There’s a lot of recurrency—the signals bounce around until it settles into an equilibrium. For example in this model, there’s a forward pass from the input data, and that activates some generative models that seem plausible. But it may be multiple models that are mutually inconsistent. For example, in the vision system, “Yarn” and “Yam” are sufficiently close that a feedforward pass would activate both possibilities simultaneously. Then there’s this message-passing algorithm where the different possibilities compete to explain the data, and it settles on one particular compositional generative model.
So this seems like a generally pretty slow inference algorithm. But the slowness is worth it, because the neocortex winds up understanding the input data, i.e. fitting it into its structured model of the world, and hence it can now flexibly query it, make predictions, etc.
I think the cerebellum is much simpler than that, and closer to an actual lookup table, and hence presumably much faster.
The cerebellum is also closer in proximity to the spinal cord, which reduces communication delays when reading proprioceptive nerves and commanding muscles.
That paper says “it has been the dominant working assumption in the field that the CS is transmitted to the cerebellar cortex via the mossy fibres (mf) and parallel fibres (pf) whereas information about the US is provided by climbing fibres (cf) originating in the inferior olive”, which is what I said (US=shoulda, CS=context). Where are you disagreeing here? Or do they contradict that later in the paper?
How does the neocortex carry the “shoulda” signal?
The neocortex is a fancy algorithm that understands the world and takes intelligent actions. It’s not perfect, but it’s pretty great! So whatever the neocortex does, that’s kinda a “ground truth” for the question of “What is the right thing to do right now?”—at least, it’s a ground truth from the perspective of the much stupider cerebellum. So my proposal is that whatever the neocortex is outputting, that’s a “shoulda” signal that the cerebellum wants to imitate.
This suggests that the neocortex can learn the cerebellar mapping and short-circuit to use it? Why does it need to go through the cerebellum to do this? Rather than via the motor cortex and efferent connections back to the muscles?
I’m not sure I understand your question. The neocortex outputs don’t need to go through the cerebellum. People can be born without a cerebellum entirely, they turn out OK. But since the cerebellum is like a super-fast memoizer / lookup table, I think the neocortex can work better by passing signals through the cerebellum.
Anyway, that was all just casual speculation, I don’t know how the motor cortex, midbrain, cerebellum, and outgoing nerves are wired together. (I’m very interested to learn, just haven’t gotten around to it.)
Hmm, this page suggests that there are both motor-pathways-through-the-cerebellum and motor-pathways-bypassing-the-cerebellum. But that’s not a reliable source—that page seems to be riddled with errors. So I dunno.
This piece is super interesting, especially the toy models.
A few clarifying questions:
-- Why does it need to be its own separate module? Can you expand on this? And even if separate modules are useful (as per your toy models and different inputs, couldn’t the neocortex also be running lookup table like auto or hetero-associative learning).
-- Can you cite this? I have seen evidence this is the case but also that the context actually comes through the climbing fibers and training (shoulda) signal through the mossy/parallel fibers. Eg here for eyeblink operant conditioning https://www.cs.cmu.edu/afs/cs/academic/class/15883-f17/readings/hesslow-2013.pdf
Can you explain how the accelerator works in more detail (esp as you use it in the later body and cognition toy models 5 and 6)? Why is the cerebellum faster at producing outputs than the neocortex? How does the neocortex carry the “shoulda” signal? Finally I’m confused by this line:
-- This suggests that the neocortex can learn the cerebellar mapping and short-circuit to use it? Why does it need to go through the cerebellum to do this? Rather than via the motor cortex and efferent connections back to the muscles?
Thank you!
Thanks for the great questions!!
Maybe you’re ahead of me, but it took me until long after this post—just a couple weeks ago—to realize that you can take a neural circuit set up for RL, and jury-rig it to do supervised learning instead.
I think this is a big part of the story behind what the vmPFC is doing. And, in a certain sense, the amygdala too. More on this in a forthcoming post.
I think of the neocortex as doing analysis-by-synthesis—it searches through a space of generative models for one that matches the input data. There’s a lot of recurrency—the signals bounce around until it settles into an equilibrium. For example in this model, there’s a forward pass from the input data, and that activates some generative models that seem plausible. But it may be multiple models that are mutually inconsistent. For example, in the vision system, “Yarn” and “Yam” are sufficiently close that a feedforward pass would activate both possibilities simultaneously. Then there’s this message-passing algorithm where the different possibilities compete to explain the data, and it settles on one particular compositional generative model.
So this seems like a generally pretty slow inference algorithm. But the slowness is worth it, because the neocortex winds up understanding the input data, i.e. fitting it into its structured model of the world, and hence it can now flexibly query it, make predictions, etc.
I think the cerebellum is much simpler than that, and closer to an actual lookup table, and hence presumably much faster.
The cerebellum is also closer in proximity to the spinal cord, which reduces communication delays when reading proprioceptive nerves and commanding muscles.
That paper says “it has been the dominant working assumption in the field that the CS is transmitted to the cerebellar cortex via the mossy fibres (mf) and parallel fibres (pf) whereas information about the US is provided by climbing fibres (cf) originating in the inferior olive”, which is what I said (US=shoulda, CS=context). Where are you disagreeing here? Or do they contradict that later in the paper?
The neocortex is a fancy algorithm that understands the world and takes intelligent actions. It’s not perfect, but it’s pretty great! So whatever the neocortex does, that’s kinda a “ground truth” for the question of “What is the right thing to do right now?”—at least, it’s a ground truth from the perspective of the much stupider cerebellum. So my proposal is that whatever the neocortex is outputting, that’s a “shoulda” signal that the cerebellum wants to imitate.
I’m not sure I understand your question. The neocortex outputs don’t need to go through the cerebellum. People can be born without a cerebellum entirely, they turn out OK. But since the cerebellum is like a super-fast memoizer / lookup table, I think the neocortex can work better by passing signals through the cerebellum.
Anyway, that was all just casual speculation, I don’t know how the motor cortex, midbrain, cerebellum, and outgoing nerves are wired together. (I’m very interested to learn, just haven’t gotten around to it.)
Hmm, this page suggests that there are both motor-pathways-through-the-cerebellum and motor-pathways-bypassing-the-cerebellum. But that’s not a reliable source—that page seems to be riddled with errors. So I dunno.