surely the plan to eat must originate in the cortex like every other plan, but it sure feels like it’s tied into the hypothalamus in some really important way
One thing is: I think you’re assuming a parallel model of decision-making—all plans are proposed in parallel, and the striatum picks a winner.
My scheme does have that, but then it also has a serial part: you consider one plan, then the next plan, etc. And each time you switch plans, there’s a dopamine signal that says whether this new plan is better or worse than the status quo / previous plan.
I think there’s good evidence for partially-serial consideration of options, at least in primates (e.g. Fig. 2b here). I mean, that’s obvious from introspection. My hunch is that partially-serial decision-making is universal in vertebrates.
Like, imagine the lamprey is swimming towards place A, and it gets to a fork where it could instead turn and go to place B. I think “the idea of going to place B” pops into the lamprey’s brain (pallium), displacing the old plan, at least for a moment. Then a dopamine signal promptly appears that says whether this new plan is better or worse than the old plan. If it’s worse (dopamine pause), the lamprey continues along its original trajectory without missing a beat. This is partially-serial decision-making. I don’t know how else the system could possibly work. Different pallium location memories are (at least partially) made out of the activations of different sparse subsets of neurons from the same pool of neurons, I think. You just can’t activate a bunch of them at once, it wouldn’t work, they would interfere with each other, AFAICT.
Anyway, if options are considered serially, things become simpler. All you really need is a mechanism for the hypothalamus to guess “if we do the current plan, how much and what type of food will I eat?”. (Such a mechanism does seem to exist AFAICT—in fact, I think mammals have two such mechanisms!)
OK, so then imagine a back-and-forth dialog.
The neocortex proposes a plan.
The hypothalamus & brainstem say “I’m hungry, but I notice that this plan won’t lead to eating any food. Bzzzz, Rejected! Try again.”
The neocortex proposes a different plan.
…Etc. etc.
(And eventually, the cortex learns that when the body is hungry, maybe don’t even bother proposing plans that won’t involve eating!)
Base thoughts seem like literally animalistic desires
I’m happy for you to intuitively think of “the desire to eat” as an “animalistic instinct”. I guess I’m encouraging you to also intuitively think of things like “the desire to be well-respected by the people whom I myself most respect” as being also an “animalistic instinct”.
The thing is, everything that comes out of the latter instinct is highly ego-syntonic, so we tend to unthinkingly identify with them, instead of externalizing them, I think.
For example, if I binge-eat, I’m happy to say to myself “my brainstem made me do it”. Whereas if I do something that delights all my favorite in-group people, I would find it deeply threatening and insulting to say to myself “my brainstem made me do it”.
It feels like on stimulants, I have more “willpower” : it’s easy to take the “noble” choice when it might otherwise be hard. Likewise, when I’m drunk...
I think that the brainstem always penalizes (subtracts points from) plans that are anticipated to entail mental concentration. I have no idea why this is the case (evolutionarily)—maybe it’s something about energy, or opportunity cost, or the tendency to not notice lions sneaking up behind us because we’re so lost in thought. Beats me.
And I think that the more tired you are, the bigger the penalty that the brainstem applies to plans that entail mental concentration. (Just like physical exertion.)
(BTW this is my version of what you described as “motionlessness prior”.)
I think this impacts willpower in two ways:
First, we are often “applying willpower” towards things that entail mental concentration and/or physical exertion, like homework and exercise. So the more tired we are, the more brainstem skepticism we have to overcome. So we’re more likely to fail.
Second, the very process of trying to “apply willpower” (= reframing plans to pass muster with the brainstem while still satisfying various constraints) itself requires mental concentration. So if you’re sufficiently tired, the brainstem will veto even the idea of trying to “apply willpower”, and also be quicker to shut down the process if it’s not immediately successful.
(Same for feeling sick or stressed. And the opposite for being on stimulants.)
(As for being drunk, we have both those items plus a third item, which is that “applying willpower” requires mental concentration, and maybe a sufficiently drunk neocortex is just plain incapable of concentrating on anything in particular, for reasons unrelated to motivation.)
Are thoughts about why I should stay home really less rewarding than thoughts about why I should go to the gym? I’m imagining the opposite
Let’s suppose that you do in fact find it more pleasant to stay home in bed than go to the gym. And let’s further suppose that you eventually wind up going to the gym anyway. As an onlooker, I want to ask the question: Why did you do that? There must have been something appealing to you about going to the gym, or else you wouldn’t have done it. People don’t do unpleasant things for no reason whatsoever.
So if “the act of exercising” is less pleasant to you than “the act of staying home” … and yet I just saw you drive off to the gym … I think that the only explanation is: the thought of “I will have exercised” is much more appealing to you than “I will have stayed home”—so much so that it more-than-compensates for the immediate aversiveness.
(Or if the motivation is “if I stay home I’ll think of myself as a fat slob”, we can say “much less unappealing” instead of “much more appealing”.)
So I stand by the asymmetry. In terms of immediately salient aspects, staying at home is more desirable. In terms of less salient aspects, like all the consequences and implications that you think of when you hold each of the plans in your mind, going to the gym is more desirable (or less undesirable), at least for the person who actually winds up going.
Base motivations also seem like things which have a more concrete connection to reinforcement learning.
I think “noble motivations” are often ultimately rooted in our social instincts (“everyone I respect is talking about how great X is, I want to do X too.”)
I haven’t seen a satisfying (to me) gears-level explanation of social instincts that relates them to RL and everything else we know about the brain. I aspire to rectify this someday… I have vague ideas, but it’s a work in progress. :-)
In terms of how long or short the RL loop is, things like imagination and memory can bridge gaps in time. Like, if all the cool kids talk about how getting ripped is awesome, then going to the gym can be immediately rewarding, because while you’re there, you’re imagining the cool kids being impressed by your muscles two months from now. Or in the other direction: two months later the cool kids are impressed by your muscles, and you remember how you’re ripped because you went to the gym, and then your brain flags the concept of “going to the gym” as a thing that leads to praise. I think the brain’s RL system can handle those kinds of things.
someone who knows planes are very safe but is scared of flying anyway.
I think one of the “assessment calculators” (probably in the amygdala) is guessing which plans will lead to mortal danger, and this calculator is giving very high scores to any plan that involves flying in planes. We (= high-level planning cortex) don’t have conscious control over any of these assessment calculators. (Evolution designed it like that for good reason, to avoid wireheading.) The best we can do is try to nudge things on the margin by framing plans in different ways, attending to different aspects of them, etc. (And of course we can change the assessment calculators through experience—e.g. exposure therapy.)
wanting to scratch an itch while meditating
I think there’s a special thing for itches—along with getting poked on the shoulder, flashing lights, sudden sounds, pains, etc. I think the brainstem detects these things directly (superior colliculus etc.) and then “forces” the high-level-planning part of the neocortex (global neuronal workspace or whatever) to pay attention to them.
So if you get poked, the brainstem forces the high-level planner to direct attention towards the relevant part of somatosensory cortex. If there’s a flashing light, the brainstem forces attention towards the corresponding part of visual cortex. If there’s an itch, I think it’s interoceptive cortex (in the insula), etc.
Not sure but I think the mechanism for this involves PPN sending acetylcholine to the corresponding sensory cortex.
Basically, it doesn’t matter whether the current top-down model is “trying” to make strong predictions or weak predictions about that part of the sensory field. The exogenous injection of acetylcholine simply overrules it. So you’re more-or-less forced to keep thinking about the itch. Any other thoughts tend to get destabilized. (Dammit, evil brainstem!)
The effort doesn’t feel like thinking of new framings
I don’t think it has to…
What is “effort”, objectively? Maybe something like (1) you’re doing something, (2) it entails mental concentration or physical exertion (which as mentioned above are more penalized when you’re tired), (3) doing it is causing unpleasant feelings.
Well, that fits the itch example! The thing I’m doing is “thinking a certain thought” (namely, a thought that involves not scratching my itch). It entails a great feat of mental concentration—thanks to that acetylcholine thing I mentioned above, constantly messing with my attention. And every second that I continue to think that thought causes an unpleasant itching sensation to continue. So, maybe that’s all the ingredients we need for it to feel like “effort”.
(A plan can get a high brainstem reward while also feeling unpleasant.)
wanting to meditate well has a long chain of logical explanations.
Hmm, I don’t think that’s relevant. Imagine you had an itch where every time you scratch it, it makes your whole body hurt really bad for 30 seconds. And then it starts itching again. You would be in the same place as the meditation example, “exerting willpower” not to scratch the itch. Right? Sorry if I’m missing your point here.
Thanks for your helpful comments!!! :)
One thing is: I think you’re assuming a parallel model of decision-making—all plans are proposed in parallel, and the striatum picks a winner.
My scheme does have that, but then it also has a serial part: you consider one plan, then the next plan, etc. And each time you switch plans, there’s a dopamine signal that says whether this new plan is better or worse than the status quo / previous plan.
I think there’s good evidence for partially-serial consideration of options, at least in primates (e.g. Fig. 2b here). I mean, that’s obvious from introspection. My hunch is that partially-serial decision-making is universal in vertebrates.
Like, imagine the lamprey is swimming towards place A, and it gets to a fork where it could instead turn and go to place B. I think “the idea of going to place B” pops into the lamprey’s brain (pallium), displacing the old plan, at least for a moment. Then a dopamine signal promptly appears that says whether this new plan is better or worse than the old plan. If it’s worse (dopamine pause), the lamprey continues along its original trajectory without missing a beat. This is partially-serial decision-making. I don’t know how else the system could possibly work. Different pallium location memories are (at least partially) made out of the activations of different sparse subsets of neurons from the same pool of neurons, I think. You just can’t activate a bunch of them at once, it wouldn’t work, they would interfere with each other, AFAICT.
Anyway, if options are considered serially, things become simpler. All you really need is a mechanism for the hypothalamus to guess “if we do the current plan, how much and what type of food will I eat?”. (Such a mechanism does seem to exist AFAICT—in fact, I think mammals have two such mechanisms!)
OK, so then imagine a back-and-forth dialog.
The neocortex proposes a plan.
The hypothalamus & brainstem say “I’m hungry, but I notice that this plan won’t lead to eating any food. Bzzzz, Rejected! Try again.”
The neocortex proposes a different plan.
…Etc. etc.
(And eventually, the cortex learns that when the body is hungry, maybe don’t even bother proposing plans that won’t involve eating!)
I’m happy for you to intuitively think of “the desire to eat” as an “animalistic instinct”. I guess I’m encouraging you to also intuitively think of things like “the desire to be well-respected by the people whom I myself most respect” as being also an “animalistic instinct”.
The thing is, everything that comes out of the latter instinct is highly ego-syntonic, so we tend to unthinkingly identify with them, instead of externalizing them, I think.
For example, if I binge-eat, I’m happy to say to myself “my brainstem made me do it”. Whereas if I do something that delights all my favorite in-group people, I would find it deeply threatening and insulting to say to myself “my brainstem made me do it”.
I think that the brainstem always penalizes (subtracts points from) plans that are anticipated to entail mental concentration. I have no idea why this is the case (evolutionarily)—maybe it’s something about energy, or opportunity cost, or the tendency to not notice lions sneaking up behind us because we’re so lost in thought. Beats me.
And I think that the more tired you are, the bigger the penalty that the brainstem applies to plans that entail mental concentration. (Just like physical exertion.)
(BTW this is my version of what you described as “motionlessness prior”.)
I think this impacts willpower in two ways:
First, we are often “applying willpower” towards things that entail mental concentration and/or physical exertion, like homework and exercise. So the more tired we are, the more brainstem skepticism we have to overcome. So we’re more likely to fail.
Second, the very process of trying to “apply willpower” (= reframing plans to pass muster with the brainstem while still satisfying various constraints) itself requires mental concentration. So if you’re sufficiently tired, the brainstem will veto even the idea of trying to “apply willpower”, and also be quicker to shut down the process if it’s not immediately successful.
(Same for feeling sick or stressed. And the opposite for being on stimulants.)
(As for being drunk, we have both those items plus a third item, which is that “applying willpower” requires mental concentration, and maybe a sufficiently drunk neocortex is just plain incapable of concentrating on anything in particular, for reasons unrelated to motivation.)
Let’s suppose that you do in fact find it more pleasant to stay home in bed than go to the gym. And let’s further suppose that you eventually wind up going to the gym anyway. As an onlooker, I want to ask the question: Why did you do that? There must have been something appealing to you about going to the gym, or else you wouldn’t have done it. People don’t do unpleasant things for no reason whatsoever.
So if “the act of exercising” is less pleasant to you than “the act of staying home” … and yet I just saw you drive off to the gym … I think that the only explanation is: the thought of “I will have exercised” is much more appealing to you than “I will have stayed home”—so much so that it more-than-compensates for the immediate aversiveness.
(Or if the motivation is “if I stay home I’ll think of myself as a fat slob”, we can say “much less unappealing” instead of “much more appealing”.)
So I stand by the asymmetry. In terms of immediately salient aspects, staying at home is more desirable. In terms of less salient aspects, like all the consequences and implications that you think of when you hold each of the plans in your mind, going to the gym is more desirable (or less undesirable), at least for the person who actually winds up going.
I think “noble motivations” are often ultimately rooted in our social instincts (“everyone I respect is talking about how great X is, I want to do X too.”)
I haven’t seen a satisfying (to me) gears-level explanation of social instincts that relates them to RL and everything else we know about the brain. I aspire to rectify this someday… I have vague ideas, but it’s a work in progress. :-)
In terms of how long or short the RL loop is, things like imagination and memory can bridge gaps in time. Like, if all the cool kids talk about how getting ripped is awesome, then going to the gym can be immediately rewarding, because while you’re there, you’re imagining the cool kids being impressed by your muscles two months from now. Or in the other direction: two months later the cool kids are impressed by your muscles, and you remember how you’re ripped because you went to the gym, and then your brain flags the concept of “going to the gym” as a thing that leads to praise. I think the brain’s RL system can handle those kinds of things.
I think one of the “assessment calculators” (probably in the amygdala) is guessing which plans will lead to mortal danger, and this calculator is giving very high scores to any plan that involves flying in planes. We (= high-level planning cortex) don’t have conscious control over any of these assessment calculators. (Evolution designed it like that for good reason, to avoid wireheading.) The best we can do is try to nudge things on the margin by framing plans in different ways, attending to different aspects of them, etc. (And of course we can change the assessment calculators through experience—e.g. exposure therapy.)
I think there’s a special thing for itches—along with getting poked on the shoulder, flashing lights, sudden sounds, pains, etc. I think the brainstem detects these things directly (superior colliculus etc.) and then “forces” the high-level-planning part of the neocortex (global neuronal workspace or whatever) to pay attention to them.
So if you get poked, the brainstem forces the high-level planner to direct attention towards the relevant part of somatosensory cortex. If there’s a flashing light, the brainstem forces attention towards the corresponding part of visual cortex. If there’s an itch, I think it’s interoceptive cortex (in the insula), etc.
Not sure but I think the mechanism for this involves PPN sending acetylcholine to the corresponding sensory cortex.
Basically, it doesn’t matter whether the current top-down model is “trying” to make strong predictions or weak predictions about that part of the sensory field. The exogenous injection of acetylcholine simply overrules it. So you’re more-or-less forced to keep thinking about the itch. Any other thoughts tend to get destabilized. (Dammit, evil brainstem!)
I don’t think it has to…
What is “effort”, objectively? Maybe something like (1) you’re doing something, (2) it entails mental concentration or physical exertion (which as mentioned above are more penalized when you’re tired), (3) doing it is causing unpleasant feelings.
Well, that fits the itch example! The thing I’m doing is “thinking a certain thought” (namely, a thought that involves not scratching my itch). It entails a great feat of mental concentration—thanks to that acetylcholine thing I mentioned above, constantly messing with my attention. And every second that I continue to think that thought causes an unpleasant itching sensation to continue. So, maybe that’s all the ingredients we need for it to feel like “effort”.
(A plan can get a high brainstem reward while also feeling unpleasant.)
Hmm, I don’t think that’s relevant. Imagine you had an itch where every time you scratch it, it makes your whole body hurt really bad for 30 seconds. And then it starts itching again. You would be in the same place as the meditation example, “exerting willpower” not to scratch the itch. Right? Sorry if I’m missing your point here.