So, as our first example, suppose my brain tends to assign a positive valence to thoughts of Tom Hanks. I might say “Tom Hanks is really great” or “I like Tom Hanks” or “I have a lot of respect for Tom Hanks”. My brain would apply the Halo Effect to him (see §3.4.4) and I’m more inclined to give the benefit of the doubt to anything having to do with him (see §2.4.3).
Why would your brain assign positive valence to these thoughts? These thoughts about persons specifically, beyond what it learns of what thinking and speaking and acting related to them entail?
I’m not saying the brain can’t reward such thoughts. It can for sure learn what people are and that people can have an outsized influence on the success of its actions. But I want to drill down on what it would mean if the brain would selectively reward such thoughts. I think there is a risk of self-reinforcing or circularity. Thinking of people is rewarding because thinking of people is warding, so do more of it.
If, when lots of people think of Person X, their brains tend to respond with fear of them (or awe, or something in that vicinity) then that fact is related to X’s dominance.
I notice that you do not use the symmetrical expression “respond with negative valence”. Is that because there is no antagonistic “negative valence” neurotransmitter? I know that there are neurotransmitters for stress but it seems that antagonistic learning works differently in the brain than dopamine reward, or?
Why would your brain assign positive valence to these thoughts? These thoughts about persons specifically, beyond what it learns of what thinking and speaking and acting related to them entail?
I feel like you could ask the same question about any valence-of-a-concept (§2.4). Why would your brain assign positive valence to “democracy”, beyond what it learns of what thinking and speaking and acting related to democracy entails? And my answer is: it’s a learned normative heuristic (§2.4.3). By the same token, if, on many occasions, I find myself having a good time in Tom-Hanks-associated movies, impressing my friends with Tom-Hanks-associated witticisms, etc., so my brain winds up with a more general heuristic that I should be motivated by Tom-Hanks-involving things (with the caveats in §2.4.1.1), and then it applies that normative heuristic to novel situations like “other things equal I want to buy a Tom Hanks action figure” or “other things equal, if I learn that Tom Hanks does X, I should update in the direction of X being a good idea”.
I think there is a risk of self-reinforcing or circularity. Thinking of people is rewarding because thinking of people is warding, so do more of it.
There’s an “inference algorithm” (what the brain should do right now) and there’s a “learning algorithm” (how the brain should self-modify so as to be more effective in the future). I’ve been focusing almost exclusively on the inference algorithm in this series. Your comment here is kinda mixing up the inference algorithm and learning algorithm in a weird-to-me way. Like, if my brain right now is assigning positive valence to thoughts-involving-Tom-Hanks, then I will find such thoughts and ideas motivating right now (inference algorithm), but my brain won’t necessarily update those thoughts and ideas to be even more motivating in the future (learning algorithm). That would depend on the error signal going into the learning algorithm, which is a different thing and outside the scope of this series. Does that help?
I notice that you do not use the symmetrical expression “respond with negative valence”. Is that because there is no antagonistic “negative valence” neurotransmitter? I know that there are neurotransmitters for stress but it seems that antagonistic learning works differently in the brain than dopamine reward, or?
OK, the more complete version would be
If, when lots of people think of Person X, their brains tend to respond with positive valence, then that fact is related to X having high social status / prestige (as above);
If, when lots of people think of Person X, their brains tend to respond with negative valence, then that fact is related to X having low social status / prestige (as above);
If, when lots of people think of Person X, their brains tend to respond with fear of them (or awe, or something in that vicinity) then that fact is related to X having high dominance.
If, when lots of people think of Person X, their brains tend to respond with lack-of-fear of them (or lack-of-awe, or feeling-of-safety, or feeling-of-the-stakes-being-low, or something in that vicinity) then that fact is related to X having low dominance.
I’m thinking of valence as a signal that can swing both positive and negative, as opposed to a positive-valence signal in the brain and a separate negative-valence signal in the brain. I acknowledge that there are areas in the brain where positive and negative contributions get calculated separately, but I think those pathways mostly get merged back together (positive minus negative) into a final all-things-considered valence signal.
Wait, do you mean this abstractly, or in the sense that there is a signal in the brain (a neurotransmitter etc.) that can be both positive and negative? How would that work biophysically?
A neuron can have a normal baseline of activity, and then be more-active-than-baseline sometimes and less-active-than-baseline other times. Phasic dopamine is the famous example that I especially have in mind here (cf. “dopamine pause” versus “dopamine burst”). I presume there are other examples too, but that’s something of a guess.
Here’s another option: If Signal X inhibits Neuron Group Z, and Signal Y excites Neuron Group Z, then we can abstractly subtract Signal X from Signal Y to get an abstract signal that can swing both positive and negative, and have coherent effects in the brain of either sign, even if there isn’t any one physical signal corresponding to that.
Do I understand correctly that the baseline of tonic dopamine sets the zero point and the bursts or absence of bursts indicate the positive/negative signal around that zero point?
The second option also makes general sense to me. It would more reliably result in absolute reward signals as the zero point is indeed the absence of either signal. I wonder if there are such neuron groups that do have such inhibition properties as you propose.
Do I understand correctly that the baseline of tonic dopamine sets the zero point and the bursts or absence of bursts indicate the positive/negative signal around that zero point?
Yes, that is what I was saying, except “pause” is different from “absence of burst”. E.g.
Electrical activity of midbrain DA neurons in vivo is characterized by tonic background activity in a narrow frequency range (ca. 1-8Hz) interrupted by either transient (i.e. phasic, <500ms) sequences of high-frequency firing (>15Hz), so called “bursts”, or transient pauses of electrical activity, where DA neurons generate no action potentials.
So 1-8Hz is the baseline / zero point, a “burst” is more activity than baseline, and a “pause” is less activity than baseline.
There’s a beginner-friendly discussion of dopamine neurons in Brian Christian’s Alignment Problem book, if memory serves.
I wonder if there are such neuron groups that do have such inhibition properties as you propose.
There are inhibitory synapses all over the brain—probably a comparable number to excitatory synapses (I don’t know the exact ratio offhand). Inhibitory synapses often (maybe “usually”) use GABA as the neurotransmitter, while excitatory synapses often use glutamate as the neurotransmitter.
1-8Hz is quite a range. I guess the base rate is not stable even within an individual. That would imply that the zero point is not actually fix. Given that tge brain can also achieve the effect with inhibitory synapses, I wonder whether there is a reason for this.
I also wonder what would happen if existing neuronal networks were trained with varying zero points. Would that improve learning like dropout is improving generalization?
Some thought on 4.3
Why would your brain assign positive valence to these thoughts? These thoughts about persons specifically, beyond what it learns of what thinking and speaking and acting related to them entail?
I’m not saying the brain can’t reward such thoughts. It can for sure learn what people are and that people can have an outsized influence on the success of its actions. But I want to drill down on what it would mean if the brain would selectively reward such thoughts. I think there is a risk of self-reinforcing or circularity. Thinking of people is rewarding because thinking of people is warding, so do more of it.
I notice that you do not use the symmetrical expression “respond with negative valence”. Is that because there is no antagonistic “negative valence” neurotransmitter? I know that there are neurotransmitters for stress but it seems that antagonistic learning works differently in the brain than dopamine reward, or?
I feel like you could ask the same question about any valence-of-a-concept (§2.4). Why would your brain assign positive valence to “democracy”, beyond what it learns of what thinking and speaking and acting related to democracy entails? And my answer is: it’s a learned normative heuristic (§2.4.3). By the same token, if, on many occasions, I find myself having a good time in Tom-Hanks-associated movies, impressing my friends with Tom-Hanks-associated witticisms, etc., so my brain winds up with a more general heuristic that I should be motivated by Tom-Hanks-involving things (with the caveats in §2.4.1.1), and then it applies that normative heuristic to novel situations like “other things equal I want to buy a Tom Hanks action figure” or “other things equal, if I learn that Tom Hanks does X, I should update in the direction of X being a good idea”.
There’s an “inference algorithm” (what the brain should do right now) and there’s a “learning algorithm” (how the brain should self-modify so as to be more effective in the future). I’ve been focusing almost exclusively on the inference algorithm in this series. Your comment here is kinda mixing up the inference algorithm and learning algorithm in a weird-to-me way. Like, if my brain right now is assigning positive valence to thoughts-involving-Tom-Hanks, then I will find such thoughts and ideas motivating right now (inference algorithm), but my brain won’t necessarily update those thoughts and ideas to be even more motivating in the future (learning algorithm). That would depend on the error signal going into the learning algorithm, which is a different thing and outside the scope of this series. Does that help?
OK, the more complete version would be
If, when lots of people think of Person X, their brains tend to respond with positive valence, then that fact is related to X having high social status / prestige (as above);
If, when lots of people think of Person X, their brains tend to respond with negative valence, then that fact is related to X having low social status / prestige (as above);
If, when lots of people think of Person X, their brains tend to respond with fear of them (or awe, or something in that vicinity) then that fact is related to X having high dominance.
If, when lots of people think of Person X, their brains tend to respond with lack-of-fear of them (or lack-of-awe, or feeling-of-safety, or feeling-of-the-stakes-being-low, or something in that vicinity) then that fact is related to X having low dominance.
I’m thinking of valence as a signal that can swing both positive and negative, as opposed to a positive-valence signal in the brain and a separate negative-valence signal in the brain. I acknowledge that there are areas in the brain where positive and negative contributions get calculated separately, but I think those pathways mostly get merged back together (positive minus negative) into a final all-things-considered valence signal.
Wait, do you mean this abstractly, or in the sense that there is a signal in the brain (a neurotransmitter etc.) that can be both positive and negative? How would that work biophysically?
A neuron can have a normal baseline of activity, and then be more-active-than-baseline sometimes and less-active-than-baseline other times. Phasic dopamine is the famous example that I especially have in mind here (cf. “dopamine pause” versus “dopamine burst”). I presume there are other examples too, but that’s something of a guess.
Here’s another option: If Signal X inhibits Neuron Group Z, and Signal Y excites Neuron Group Z, then we can abstractly subtract Signal X from Signal Y to get an abstract signal that can swing both positive and negative, and have coherent effects in the brain of either sign, even if there isn’t any one physical signal corresponding to that.
Do I understand correctly that the baseline of tonic dopamine sets the zero point and the bursts or absence of bursts indicate the positive/negative signal around that zero point?
The second option also makes general sense to me. It would more reliably result in absolute reward signals as the zero point is indeed the absence of either signal. I wonder if there are such neuron groups that do have such inhibition properties as you propose.
Yes, that is what I was saying, except “pause” is different from “absence of burst”. E.g.
So 1-8Hz is the baseline / zero point, a “burst” is more activity than baseline, and a “pause” is less activity than baseline.
There’s a beginner-friendly discussion of dopamine neurons in Brian Christian’s Alignment Problem book, if memory serves.
There are inhibitory synapses all over the brain—probably a comparable number to excitatory synapses (I don’t know the exact ratio offhand). Inhibitory synapses often (maybe “usually”) use GABA as the neurotransmitter, while excitatory synapses often use glutamate as the neurotransmitter.
Thanks for the primer.
1-8Hz is quite a range. I guess the base rate is not stable even within an individual. That would imply that the zero point is not actually fix. Given that tge brain can also achieve the effect with inhibitory synapses, I wonder whether there is a reason for this.
I also wonder what would happen if existing neuronal networks were trained with varying zero points. Would that improve learning like dropout is improving generalization?