The midbrain dopamine centers (VTA, SNc) are traditionally “part of the basal ganglia” AND “part of the brainstem”. I think that these regions are where you find the “final answer” about whether a plan is good or bad, and that the dopamine signals from these regions can just directly shut down bad ideas.
But of course a lot of processing happens before you get to the “final answer”…
Specifically, I think there are basically three layers of “plan evaluation”:
First, you start with a bubbly soup of partially-formed thoughts in the neocortex, and the (dorsal) striatum does a quick rough guess of how promising the different pieces look, and gently suppresses the less-promising bits / enhances the more-promising bits, so that when you get a fully-formed thought, it’s at least reasonably promising.
Second, once you have a stable fully-formed thought, I think various parts of the brain (parts of prefrontal & cingulate & insular cortex, ventral striatum, amygdala, hippocampus (sorta)) score that thought along maybe dozens of genetically-hardcoded axes like “If I’m gonna do this plan, how appropriate would it be to cringe? To salivate? To release cortisol? To laugh? How much salt would I wind up eating? How much umami? Etc. etc.” (They learn to do these evaluations through experience, a.k.a. supervised learning.) And they send all those “scores” down to the hypothalamus and brainstem.
Finally, we’re at the hypothalamus & brainstem. They look at the information from the previous bullet point, and they combine that information with other information streams, like metabolic status information (if I’m hungry, a plan that involves eating gets extra points), and whether the superior colliculus thinks there’s a snake in the field-of-view, and so on. Taking all that information into account, they make the final decision as to whether the plan is good or bad, using a genetically-hardcoded algorithm.
Happy to discuss more; also, I’m still reading about this / talking to experts / etc., and I reserve the right to change my mind :)
Thanks, I read that, and while I wouldn’t say I’m completely enlightened, I feel like I have a good basis for reading it a few more times until it sinks in.
I interpret you as saying in this post: there is no fundamental difference between base and noble motivations, they’re just two different kinds of plans we can come up with and evaluate, and we resolve conflicts between them by trying to find frames in which one or the other seems better. Noble motivations seem to “require more willpower” only because we often spend more time working on coming up with positive frames for them, because this activity flatters our ego and so is inherently rewarding.
I’m still not sure I agree with this. My own base motivation here is that I posted a somewhat different model of willpower at https://astralcodexten.substack.com/p/towards-a-bayesian-theory-of-willpower , which is similar to yours except that it does keep a role for the difference between “base” and “noble” urges. I’m trying to figure out if I still want to defend it against this one, but my thoughts are something like:
- It feels like on stimulants, I have more “willpower” : it’s easy to take the “noble” choice when it might otherwise be hard. Likewise, when I’m drunk I have less ability to override base motivations with noble ones, and (although I guess I can’t prove it) this doesn’t seem like a purely cognitive effect where it’s harder for me to “remember” the important benefits of my noble motivations. The same is true of various low-energy states, eg tired, sick, stressed—I’m less likely to choose the noble motivation in all of them. This suggests to me that baser and nobler motivations are coming from different places, and stimulants strengthen (in your model) the connection between the noble-motivation-place and the striatum relative to the connection between the base-motivation-place the striatum, and alcohol/stress/etc weaken it.
- I’m skeptical of your explanation for the “asymmetry” of noble vs. base thoughts. Are thoughts about why I should stay home really less rewarding than thoughts about why I should go to the gym? I’m imagining the opposite—I imagine staying home in my nice warm bed, and this is a very pleasant thought, and accords with what I currently really want (to not go to the gym). On the other hand, thoughts about why I should go to the gym, if I were to verbalize them, would sound like “Ugh, I guess I have to consider the fat that I’ll be a fat slob if I don’t go, even though I wish I could just never have to think about that”.
- Base thoughts seem like literally animalistic desires—hunger seems basically built on top of the same kind of hunger a lizard or nematode feels. We know there are a bunch of brain areas in the hypothalamus etc that control hunger. So why shouldn’t this be ontologically different from nobler motivations that are different from lizards’? It seems perfectly sensible that eg stimulants strengthen something about the neocortex relative to whatever part of the hypothalamus is involved in hunger. I guess I’m realizing now how little I understand about hunger—surely the plan to eat must originate in the cortex like every other plan, but it sure feels like it’s tied into the hypothalamus in some really important way. I guess maybe hunger could have a plan-generator exactly like every other, which is modulated by hypothalamic connections? It still seems like “plans that need outside justification” vs. “plans that the hypothalamus will just keep active even if they’re stupid” is a potentially important dichotomy.
- Base motivations also seem like things which have a more concrete connection to reinforcement learning. There’s a really short reinforcement loop between “want to eat candy” and “wow, that was reinforcing”, and a really long (sometimes nonexistent) loop between going to the gym and anything good happening. Again, this makes me suspicious that the base motivations are “encoded” in some way that’s different from the nobler motivations and which explains why different substances can preferentially reinforce one relative to the other.
- The reasons for thinking of base motivations as more like priors, discussed in that post.
- Kind of a dumb objection, but this feels analogous to other problems where a conscious/intellectual knowledge fails to percolate to emotional centers of the brain, for example someone who knows planes are very safe but is scared of flying anyway. I’m not sure how to use your theory here to account for this situation, whereas if I had a theory that explained the plane phobia problem I feel like it would have to involve a concept of lower-level vs. higher-level systems that would be easy to plug into this problem.
- Another dumb anecdotal objection, but this isn’t how I consciously experience weakness of will. The example that comes to mind most easily is wanting to scratch an itch while meditating, even though I’m supposed to stay completely still. When I imagine my thought process while worrying about this, it doesn’t feel like trying to think up new reframings of the plan. It feels like some sensory region of the brain saying “HEY! ITCH! YOU SHOULD SCRATCH IT!” and my conscious brain trying to exert some effort to overcome that. The effort doesn’t feel like thinking of new framings, and the need for the effort persists long after every plausible new framing has been thought of. And it does seem relevant that “scratch itch” has no logical justification (it’s just a basic animal urge that would persist even if someone told you there was no biological cause of the itch and no way that not scratching it could hurt you), whereas wanting to meditate well has a long chain of logical explanations.
surely the plan to eat must originate in the cortex like every other plan, but it sure feels like it’s tied into the hypothalamus in some really important way
One thing is: I think you’re assuming a parallel model of decision-making—all plans are proposed in parallel, and the striatum picks a winner.
My scheme does have that, but then it also has a serial part: you consider one plan, then the next plan, etc. And each time you switch plans, there’s a dopamine signal that says whether this new plan is better or worse than the status quo / previous plan.
I think there’s good evidence for partially-serial consideration of options, at least in primates (e.g. Fig. 2b here). I mean, that’s obvious from introspection. My hunch is that partially-serial decision-making is universal in vertebrates.
Like, imagine the lamprey is swimming towards place A, and it gets to a fork where it could instead turn and go to place B. I think “the idea of going to place B” pops into the lamprey’s brain (pallium), displacing the old plan, at least for a moment. Then a dopamine signal promptly appears that says whether this new plan is better or worse than the old plan. If it’s worse (dopamine pause), the lamprey continues along its original trajectory without missing a beat. This is partially-serial decision-making. I don’t know how else the system could possibly work. Different pallium location memories are (at least partially) made out of the activations of different sparse subsets of neurons from the same pool of neurons, I think. You just can’t activate a bunch of them at once, it wouldn’t work, they would interfere with each other, AFAICT.
Anyway, if options are considered serially, things become simpler. All you really need is a mechanism for the hypothalamus to guess “if we do the current plan, how much and what type of food will I eat?”. (Such a mechanism does seem to exist AFAICT—in fact, I think mammals have two such mechanisms!)
OK, so then imagine a back-and-forth dialog.
The neocortex proposes a plan.
The hypothalamus & brainstem say “I’m hungry, but I notice that this plan won’t lead to eating any food. Bzzzz, Rejected! Try again.”
The neocortex proposes a different plan.
…Etc. etc.
(And eventually, the cortex learns that when the body is hungry, maybe don’t even bother proposing plans that won’t involve eating!)
Base thoughts seem like literally animalistic desires
I’m happy for you to intuitively think of “the desire to eat” as an “animalistic instinct”. I guess I’m encouraging you to also intuitively think of things like “the desire to be well-respected by the people whom I myself most respect” as being also an “animalistic instinct”.
The thing is, everything that comes out of the latter instinct is highly ego-syntonic, so we tend to unthinkingly identify with them, instead of externalizing them, I think.
For example, if I binge-eat, I’m happy to say to myself “my brainstem made me do it”. Whereas if I do something that delights all my favorite in-group people, I would find it deeply threatening and insulting to say to myself “my brainstem made me do it”.
It feels like on stimulants, I have more “willpower” : it’s easy to take the “noble” choice when it might otherwise be hard. Likewise, when I’m drunk...
I think that the brainstem always penalizes (subtracts points from) plans that are anticipated to entail mental concentration. I have no idea why this is the case (evolutionarily)—maybe it’s something about energy, or opportunity cost, or the tendency to not notice lions sneaking up behind us because we’re so lost in thought. Beats me.
And I think that the more tired you are, the bigger the penalty that the brainstem applies to plans that entail mental concentration. (Just like physical exertion.)
(BTW this is my version of what you described as “motionlessness prior”.)
I think this impacts willpower in two ways:
First, we are often “applying willpower” towards things that entail mental concentration and/or physical exertion, like homework and exercise. So the more tired we are, the more brainstem skepticism we have to overcome. So we’re more likely to fail.
Second, the very process of trying to “apply willpower” (= reframing plans to pass muster with the brainstem while still satisfying various constraints) itself requires mental concentration. So if you’re sufficiently tired, the brainstem will veto even the idea of trying to “apply willpower”, and also be quicker to shut down the process if it’s not immediately successful.
(Same for feeling sick or stressed. And the opposite for being on stimulants.)
(As for being drunk, we have both those items plus a third item, which is that “applying willpower” requires mental concentration, and maybe a sufficiently drunk neocortex is just plain incapable of concentrating on anything in particular, for reasons unrelated to motivation.)
Are thoughts about why I should stay home really less rewarding than thoughts about why I should go to the gym? I’m imagining the opposite
Let’s suppose that you do in fact find it more pleasant to stay home in bed than go to the gym. And let’s further suppose that you eventually wind up going to the gym anyway. As an onlooker, I want to ask the question: Why did you do that? There must have been something appealing to you about going to the gym, or else you wouldn’t have done it. People don’t do unpleasant things for no reason whatsoever.
So if “the act of exercising” is less pleasant to you than “the act of staying home” … and yet I just saw you drive off to the gym … I think that the only explanation is: the thought of “I will have exercised” is much more appealing to you than “I will have stayed home”—so much so that it more-than-compensates for the immediate aversiveness.
(Or if the motivation is “if I stay home I’ll think of myself as a fat slob”, we can say “much less unappealing” instead of “much more appealing”.)
So I stand by the asymmetry. In terms of immediately salient aspects, staying at home is more desirable. In terms of less salient aspects, like all the consequences and implications that you think of when you hold each of the plans in your mind, going to the gym is more desirable (or less undesirable), at least for the person who actually winds up going.
Base motivations also seem like things which have a more concrete connection to reinforcement learning.
I think “noble motivations” are often ultimately rooted in our social instincts (“everyone I respect is talking about how great X is, I want to do X too.”)
I haven’t seen a satisfying (to me) gears-level explanation of social instincts that relates them to RL and everything else we know about the brain. I aspire to rectify this someday… I have vague ideas, but it’s a work in progress. :-)
In terms of how long or short the RL loop is, things like imagination and memory can bridge gaps in time. Like, if all the cool kids talk about how getting ripped is awesome, then going to the gym can be immediately rewarding, because while you’re there, you’re imagining the cool kids being impressed by your muscles two months from now. Or in the other direction: two months later the cool kids are impressed by your muscles, and you remember how you’re ripped because you went to the gym, and then your brain flags the concept of “going to the gym” as a thing that leads to praise. I think the brain’s RL system can handle those kinds of things.
someone who knows planes are very safe but is scared of flying anyway.
I think one of the “assessment calculators” (probably in the amygdala) is guessing which plans will lead to mortal danger, and this calculator is giving very high scores to any plan that involves flying in planes. We (= high-level planning cortex) don’t have conscious control over any of these assessment calculators. (Evolution designed it like that for good reason, to avoid wireheading.) The best we can do is try to nudge things on the margin by framing plans in different ways, attending to different aspects of them, etc. (And of course we can change the assessment calculators through experience—e.g. exposure therapy.)
wanting to scratch an itch while meditating
I think there’s a special thing for itches—along with getting poked on the shoulder, flashing lights, sudden sounds, pains, etc. I think the brainstem detects these things directly (superior colliculus etc.) and then “forces” the high-level-planning part of the neocortex (global neuronal workspace or whatever) to pay attention to them.
So if you get poked, the brainstem forces the high-level planner to direct attention towards the relevant part of somatosensory cortex. If there’s a flashing light, the brainstem forces attention towards the corresponding part of visual cortex. If there’s an itch, I think it’s interoceptive cortex (in the insula), etc.
Not sure but I think the mechanism for this involves PPN sending acetylcholine to the corresponding sensory cortex.
Basically, it doesn’t matter whether the current top-down model is “trying” to make strong predictions or weak predictions about that part of the sensory field. The exogenous injection of acetylcholine simply overrules it. So you’re more-or-less forced to keep thinking about the itch. Any other thoughts tend to get destabilized. (Dammit, evil brainstem!)
The effort doesn’t feel like thinking of new framings
I don’t think it has to…
What is “effort”, objectively? Maybe something like (1) you’re doing something, (2) it entails mental concentration or physical exertion (which as mentioned above are more penalized when you’re tired), (3) doing it is causing unpleasant feelings.
Well, that fits the itch example! The thing I’m doing is “thinking a certain thought” (namely, a thought that involves not scratching my itch). It entails a great feat of mental concentration—thanks to that acetylcholine thing I mentioned above, constantly messing with my attention. And every second that I continue to think that thought causes an unpleasant itching sensation to continue. So, maybe that’s all the ingredients we need for it to feel like “effort”.
(A plan can get a high brainstem reward while also feeling unpleasant.)
wanting to meditate well has a long chain of logical explanations.
Hmm, I don’t think that’s relevant. Imagine you had an itch where every time you scratch it, it makes your whole body hurt really bad for 30 seconds. And then it starts itching again. You would be in the same place as the meditation example, “exerting willpower” not to scratch the itch. Right? Sorry if I’m missing your point here.
Good question!
Yes I can link it but it’s very long, sorry: Big Picture Of Phasic Dopamine.
The midbrain dopamine centers (VTA, SNc) are traditionally “part of the basal ganglia” AND “part of the brainstem”. I think that these regions are where you find the “final answer” about whether a plan is good or bad, and that the dopamine signals from these regions can just directly shut down bad ideas.
But of course a lot of processing happens before you get to the “final answer”…
Specifically, I think there are basically three layers of “plan evaluation”:
First, you start with a bubbly soup of partially-formed thoughts in the neocortex, and the (dorsal) striatum does a quick rough guess of how promising the different pieces look, and gently suppresses the less-promising bits / enhances the more-promising bits, so that when you get a fully-formed thought, it’s at least reasonably promising.
Second, once you have a stable fully-formed thought, I think various parts of the brain (parts of prefrontal & cingulate & insular cortex, ventral striatum, amygdala, hippocampus (sorta)) score that thought along maybe dozens of genetically-hardcoded axes like “If I’m gonna do this plan, how appropriate would it be to cringe? To salivate? To release cortisol? To laugh? How much salt would I wind up eating? How much umami? Etc. etc.” (They learn to do these evaluations through experience, a.k.a. supervised learning.) And they send all those “scores” down to the hypothalamus and brainstem.
Finally, we’re at the hypothalamus & brainstem. They look at the information from the previous bullet point, and they combine that information with other information streams, like metabolic status information (if I’m hungry, a plan that involves eating gets extra points), and whether the superior colliculus thinks there’s a snake in the field-of-view, and so on. Taking all that information into account, they make the final decision as to whether the plan is good or bad, using a genetically-hardcoded algorithm.
Happy to discuss more; also, I’m still reading about this / talking to experts / etc., and I reserve the right to change my mind :)
Thanks, I read that, and while I wouldn’t say I’m completely enlightened, I feel like I have a good basis for reading it a few more times until it sinks in.
I interpret you as saying in this post: there is no fundamental difference between base and noble motivations, they’re just two different kinds of plans we can come up with and evaluate, and we resolve conflicts between them by trying to find frames in which one or the other seems better. Noble motivations seem to “require more willpower” only because we often spend more time working on coming up with positive frames for them, because this activity flatters our ego and so is inherently rewarding.
I’m still not sure I agree with this. My own base motivation here is that I posted a somewhat different model of willpower at https://astralcodexten.substack.com/p/towards-a-bayesian-theory-of-willpower , which is similar to yours except that it does keep a role for the difference between “base” and “noble” urges. I’m trying to figure out if I still want to defend it against this one, but my thoughts are something like:
- It feels like on stimulants, I have more “willpower” : it’s easy to take the “noble” choice when it might otherwise be hard. Likewise, when I’m drunk I have less ability to override base motivations with noble ones, and (although I guess I can’t prove it) this doesn’t seem like a purely cognitive effect where it’s harder for me to “remember” the important benefits of my noble motivations. The same is true of various low-energy states, eg tired, sick, stressed—I’m less likely to choose the noble motivation in all of them. This suggests to me that baser and nobler motivations are coming from different places, and stimulants strengthen (in your model) the connection between the noble-motivation-place and the striatum relative to the connection between the base-motivation-place the striatum, and alcohol/stress/etc weaken it.
- I’m skeptical of your explanation for the “asymmetry” of noble vs. base thoughts. Are thoughts about why I should stay home really less rewarding than thoughts about why I should go to the gym? I’m imagining the opposite—I imagine staying home in my nice warm bed, and this is a very pleasant thought, and accords with what I currently really want (to not go to the gym). On the other hand, thoughts about why I should go to the gym, if I were to verbalize them, would sound like “Ugh, I guess I have to consider the fat that I’ll be a fat slob if I don’t go, even though I wish I could just never have to think about that”.
- Base thoughts seem like literally animalistic desires—hunger seems basically built on top of the same kind of hunger a lizard or nematode feels. We know there are a bunch of brain areas in the hypothalamus etc that control hunger. So why shouldn’t this be ontologically different from nobler motivations that are different from lizards’? It seems perfectly sensible that eg stimulants strengthen something about the neocortex relative to whatever part of the hypothalamus is involved in hunger. I guess I’m realizing now how little I understand about hunger—surely the plan to eat must originate in the cortex like every other plan, but it sure feels like it’s tied into the hypothalamus in some really important way. I guess maybe hunger could have a plan-generator exactly like every other, which is modulated by hypothalamic connections? It still seems like “plans that need outside justification” vs. “plans that the hypothalamus will just keep active even if they’re stupid” is a potentially important dichotomy.
- Base motivations also seem like things which have a more concrete connection to reinforcement learning. There’s a really short reinforcement loop between “want to eat candy” and “wow, that was reinforcing”, and a really long (sometimes nonexistent) loop between going to the gym and anything good happening. Again, this makes me suspicious that the base motivations are “encoded” in some way that’s different from the nobler motivations and which explains why different substances can preferentially reinforce one relative to the other.
- The reasons for thinking of base motivations as more like priors, discussed in that post.
- Kind of a dumb objection, but this feels analogous to other problems where a conscious/intellectual knowledge fails to percolate to emotional centers of the brain, for example someone who knows planes are very safe but is scared of flying anyway. I’m not sure how to use your theory here to account for this situation, whereas if I had a theory that explained the plane phobia problem I feel like it would have to involve a concept of lower-level vs. higher-level systems that would be easy to plug into this problem.
- Another dumb anecdotal objection, but this isn’t how I consciously experience weakness of will. The example that comes to mind most easily is wanting to scratch an itch while meditating, even though I’m supposed to stay completely still. When I imagine my thought process while worrying about this, it doesn’t feel like trying to think up new reframings of the plan. It feels like some sensory region of the brain saying “HEY! ITCH! YOU SHOULD SCRATCH IT!” and my conscious brain trying to exert some effort to overcome that. The effort doesn’t feel like thinking of new framings, and the need for the effort persists long after every plausible new framing has been thought of. And it does seem relevant that “scratch itch” has no logical justification (it’s just a basic animal urge that would persist even if someone told you there was no biological cause of the itch and no way that not scratching it could hurt you), whereas wanting to meditate well has a long chain of logical explanations.
Thanks for your helpful comments!!! :)
One thing is: I think you’re assuming a parallel model of decision-making—all plans are proposed in parallel, and the striatum picks a winner.
My scheme does have that, but then it also has a serial part: you consider one plan, then the next plan, etc. And each time you switch plans, there’s a dopamine signal that says whether this new plan is better or worse than the status quo / previous plan.
I think there’s good evidence for partially-serial consideration of options, at least in primates (e.g. Fig. 2b here). I mean, that’s obvious from introspection. My hunch is that partially-serial decision-making is universal in vertebrates.
Like, imagine the lamprey is swimming towards place A, and it gets to a fork where it could instead turn and go to place B. I think “the idea of going to place B” pops into the lamprey’s brain (pallium), displacing the old plan, at least for a moment. Then a dopamine signal promptly appears that says whether this new plan is better or worse than the old plan. If it’s worse (dopamine pause), the lamprey continues along its original trajectory without missing a beat. This is partially-serial decision-making. I don’t know how else the system could possibly work. Different pallium location memories are (at least partially) made out of the activations of different sparse subsets of neurons from the same pool of neurons, I think. You just can’t activate a bunch of them at once, it wouldn’t work, they would interfere with each other, AFAICT.
Anyway, if options are considered serially, things become simpler. All you really need is a mechanism for the hypothalamus to guess “if we do the current plan, how much and what type of food will I eat?”. (Such a mechanism does seem to exist AFAICT—in fact, I think mammals have two such mechanisms!)
OK, so then imagine a back-and-forth dialog.
The neocortex proposes a plan.
The hypothalamus & brainstem say “I’m hungry, but I notice that this plan won’t lead to eating any food. Bzzzz, Rejected! Try again.”
The neocortex proposes a different plan.
…Etc. etc.
(And eventually, the cortex learns that when the body is hungry, maybe don’t even bother proposing plans that won’t involve eating!)
I’m happy for you to intuitively think of “the desire to eat” as an “animalistic instinct”. I guess I’m encouraging you to also intuitively think of things like “the desire to be well-respected by the people whom I myself most respect” as being also an “animalistic instinct”.
The thing is, everything that comes out of the latter instinct is highly ego-syntonic, so we tend to unthinkingly identify with them, instead of externalizing them, I think.
For example, if I binge-eat, I’m happy to say to myself “my brainstem made me do it”. Whereas if I do something that delights all my favorite in-group people, I would find it deeply threatening and insulting to say to myself “my brainstem made me do it”.
I think that the brainstem always penalizes (subtracts points from) plans that are anticipated to entail mental concentration. I have no idea why this is the case (evolutionarily)—maybe it’s something about energy, or opportunity cost, or the tendency to not notice lions sneaking up behind us because we’re so lost in thought. Beats me.
And I think that the more tired you are, the bigger the penalty that the brainstem applies to plans that entail mental concentration. (Just like physical exertion.)
(BTW this is my version of what you described as “motionlessness prior”.)
I think this impacts willpower in two ways:
First, we are often “applying willpower” towards things that entail mental concentration and/or physical exertion, like homework and exercise. So the more tired we are, the more brainstem skepticism we have to overcome. So we’re more likely to fail.
Second, the very process of trying to “apply willpower” (= reframing plans to pass muster with the brainstem while still satisfying various constraints) itself requires mental concentration. So if you’re sufficiently tired, the brainstem will veto even the idea of trying to “apply willpower”, and also be quicker to shut down the process if it’s not immediately successful.
(Same for feeling sick or stressed. And the opposite for being on stimulants.)
(As for being drunk, we have both those items plus a third item, which is that “applying willpower” requires mental concentration, and maybe a sufficiently drunk neocortex is just plain incapable of concentrating on anything in particular, for reasons unrelated to motivation.)
Let’s suppose that you do in fact find it more pleasant to stay home in bed than go to the gym. And let’s further suppose that you eventually wind up going to the gym anyway. As an onlooker, I want to ask the question: Why did you do that? There must have been something appealing to you about going to the gym, or else you wouldn’t have done it. People don’t do unpleasant things for no reason whatsoever.
So if “the act of exercising” is less pleasant to you than “the act of staying home” … and yet I just saw you drive off to the gym … I think that the only explanation is: the thought of “I will have exercised” is much more appealing to you than “I will have stayed home”—so much so that it more-than-compensates for the immediate aversiveness.
(Or if the motivation is “if I stay home I’ll think of myself as a fat slob”, we can say “much less unappealing” instead of “much more appealing”.)
So I stand by the asymmetry. In terms of immediately salient aspects, staying at home is more desirable. In terms of less salient aspects, like all the consequences and implications that you think of when you hold each of the plans in your mind, going to the gym is more desirable (or less undesirable), at least for the person who actually winds up going.
I think “noble motivations” are often ultimately rooted in our social instincts (“everyone I respect is talking about how great X is, I want to do X too.”)
I haven’t seen a satisfying (to me) gears-level explanation of social instincts that relates them to RL and everything else we know about the brain. I aspire to rectify this someday… I have vague ideas, but it’s a work in progress. :-)
In terms of how long or short the RL loop is, things like imagination and memory can bridge gaps in time. Like, if all the cool kids talk about how getting ripped is awesome, then going to the gym can be immediately rewarding, because while you’re there, you’re imagining the cool kids being impressed by your muscles two months from now. Or in the other direction: two months later the cool kids are impressed by your muscles, and you remember how you’re ripped because you went to the gym, and then your brain flags the concept of “going to the gym” as a thing that leads to praise. I think the brain’s RL system can handle those kinds of things.
I think one of the “assessment calculators” (probably in the amygdala) is guessing which plans will lead to mortal danger, and this calculator is giving very high scores to any plan that involves flying in planes. We (= high-level planning cortex) don’t have conscious control over any of these assessment calculators. (Evolution designed it like that for good reason, to avoid wireheading.) The best we can do is try to nudge things on the margin by framing plans in different ways, attending to different aspects of them, etc. (And of course we can change the assessment calculators through experience—e.g. exposure therapy.)
I think there’s a special thing for itches—along with getting poked on the shoulder, flashing lights, sudden sounds, pains, etc. I think the brainstem detects these things directly (superior colliculus etc.) and then “forces” the high-level-planning part of the neocortex (global neuronal workspace or whatever) to pay attention to them.
So if you get poked, the brainstem forces the high-level planner to direct attention towards the relevant part of somatosensory cortex. If there’s a flashing light, the brainstem forces attention towards the corresponding part of visual cortex. If there’s an itch, I think it’s interoceptive cortex (in the insula), etc.
Not sure but I think the mechanism for this involves PPN sending acetylcholine to the corresponding sensory cortex.
Basically, it doesn’t matter whether the current top-down model is “trying” to make strong predictions or weak predictions about that part of the sensory field. The exogenous injection of acetylcholine simply overrules it. So you’re more-or-less forced to keep thinking about the itch. Any other thoughts tend to get destabilized. (Dammit, evil brainstem!)
I don’t think it has to…
What is “effort”, objectively? Maybe something like (1) you’re doing something, (2) it entails mental concentration or physical exertion (which as mentioned above are more penalized when you’re tired), (3) doing it is causing unpleasant feelings.
Well, that fits the itch example! The thing I’m doing is “thinking a certain thought” (namely, a thought that involves not scratching my itch). It entails a great feat of mental concentration—thanks to that acetylcholine thing I mentioned above, constantly messing with my attention. And every second that I continue to think that thought causes an unpleasant itching sensation to continue. So, maybe that’s all the ingredients we need for it to feel like “effort”.
(A plan can get a high brainstem reward while also feeling unpleasant.)
Hmm, I don’t think that’s relevant. Imagine you had an itch where every time you scratch it, it makes your whole body hurt really bad for 30 seconds. And then it starts itching again. You would be in the same place as the meditation example, “exerting willpower” not to scratch the itch. Right? Sorry if I’m missing your point here.