But as far as we can tell, our behavior is often not determined by our wanting a particular state of affairs, but by how our options are framed.
Moreover, neurons in the parietal and obritofrontal corticies encode value in a reference-dependent way — that is, they do not encode value for objective states of affairs.
I’m not certain your examples of reference-dependent encoding of sense-data really demonstrate or have much to do with a lack of objective goals. (Of course, the framing effect example demonstrates this plenty well. :P ) As you point out, this is largely just adjusting for irrelevant background, like whether the sun is out, when what we care about has nothing to do with that. This is just throwing away the information at an early stage, rather than later after having explicitly determined that it’s irrelevant to our goals.
I agree that the framing effect is more important than the reference-dependence of sense-data encoding. However, the loss of sense-data is not always just “adjusting for irrelevant background”, and is not always throwing away something we would later have decided is “irrelevant to our goals.”
When I first read the post, I thought you were going to say something along the lines of:
“Evolution has optimized us to strip away the irrelevant features when it comes to vision, since it’s been vital for our survival. But evolution hasn’t done that for things like abstract value, since there’s been no selection pressure for that. It’s bad that our judgments in cases like the K&T examples don’t work more like vision, but that’s how it goes”.
Indeed, saying “let’s make the problem worse” and then bringing up vision feels a bit weird. After all, vision seems like a case where our brain does things exactly right—it ignores the “framing effects” caused by changed lightning conditions and leaves invariant the things that actually matter.
An illuminating (no pun intended) example of when the adjustment to the ambient level of sense-data affects what people think they want would be nice. Without it the whole section seems to detract from your point.
But I’m not raising a puzzle about how people think they want things even when they are behavioristic machines. I’m raising a puzzle about how we can be said to actually want things even when they are behavioristic machines that, for example, exhibit framing effects and can’t use neurons to encode value for the objective intensities of stimuli.
Suppose you have a neurological disorder that will be cured by a 140-volt electrical shock. If your brain can’t encode value for propositions or simulated states of affairs or anything like that, but only for stimuli, then this reference point business I described means that your brain doesn’t have the option of encoding value for a 140-volt electrical shock, because it never receives that kind of information in the first place. The transducer discards information about the objective intensity of the stimuli before the signal reaches the brain.
As Kaj says, this is a smart solution to lots of problems, but it does mean that the brain cannot encode value for the objective intensity of stimuli… at least given what I’ve explained in this post so far. (Model-based representations of value will be described later.)
If the neurological problem is located in the brain, then the brain does record information about the objective intensity of the stimuli, by being cured or not cured.
I’m confused about what the purpose of this example is. There are easier ways to show why not encoding values for propositions is problematic.
If the neurological problem is located in the brain, then the brain does record information about the objective intensity of the stimuli, by being cured or not cured.
Sure, but what I’m saying is that this doesn’t happen in a way that allows your neurons to encode value for a 140-volt electrical shock. Perhaps you’ve already accepted this and find it obvious, but others (e.g. economists) do not. This kind of information about how the brain works constrains our models of human behavior, just like the stochasticity of neuron firing does.
I’m confused about what the purpose of this example is. There are easier ways to show why not encoding values for propositions is problematic.
But I’m not trying to show why encoding values for propositions is problematic. I’m trying to say that the brain does not encode values for objective intensities of stimuli.
But I’m not trying to show why encoding values for propositions is problematic. I’m trying to say that the brain does not encode values for objective intensities of stimuli.
Given that one could use propositions about objective intensities of stimuli (as you do now, to point out that what’s encoded in this particular simple way is not it), the thesis is still unclear.
Sure, but that depends on a different mechanism we don’t know much about, then. What I’m saying is that “Whaddyaknow, we discovered a mechanism that actually encodes value for stimuli with neuron firing rates! Ah, but it can’t encode value for the objective intensities of stimuli, because the brain doesn’t have that information. So that constrains our theories about the motivation of human behavior.”
Ah, but it can’t encode value for the objective intensities of stimuli, because the brain doesn’t have that information.
The brain has (some measure of reference/access to) that information, just not in that particular form. And if it has (reference to) that information, it’s not possible to conclude that motivation doesn’t refer to it. It just doesn’t refer to it through exclusively the form of representation that doesn’t have the information, but then it would be very surprising if motivation compartmentalized so.
Right. I guess I’m struggling for a concise way to say what I’m trying to say, and hoping you’ll interpret me correctly based on the long paragraphs I’ve written explaining what I mean by these shorter sentences. Maybe something like:
“Whaddyaknow, we discovered a mechanism that actually encodes value for stimuli with neuron firing rates! Ah, but this particular mechanism can’t encode value for the objective intensities of stimuli, because this mechanism discards that information at the transducer. So that constrains our theories about the motivation of human behavior.”
we discovered a mechanism that actually encodes value for stimuli with neuron firing rates!
Also, this doesn’t sound right. Why is that behavioral pattern “value”? Maybe it should be edited out of the system, like pain, or reversed, or modified in some complicated way.
Doesn’t really help. The problem is that (normative) motivation is the whole thing, particular (unusual) decisions can be formed by any component, so it’s unclear how to rule out stuff on the basis of properties of particular better-understood components.
Behavior is easier to analyze, you can see which factors contribute how much, and in this sense you can say that particular classes of behavior are determined mostly by this here mechanism that doesn’t have certain data, and so behavior is independent from that data. But such conclusions won’t generalize to normative motivation, because prevailing patterns of behavior might be suboptimal, and it’s possible to improve them (by exercising the less-prevalent modes of behavior that are less understood), making them depend on things that they presently don’t depend on.
Uh oh. How did ‘should’ sneak its way into our discussion? I’m just talking about positive accounts of human motivation.
I guess the objection I have is to calling the behavioral summary “motivation”, a term that has normative connotations (similarly, “value”, “desire”, “wants”, etc.). Asking “Do we really want X?” (as in, does a positive account of some notion of “wanting” say that we “want” X, to the best of our scientific knowledge) sounds too similar to asking “Should we pursue X?” or even “Can we pursue X?”, but is a largely unrelated question with similarly unrelated answers.
I’m using these terms the way they are standardly used in the literature. If you object to the common usage, perhaps you could just read my articles with the assumption that I’m using these words the way neuroscientists and psychologists do, and then state your concerns about the standard language in the comments? I can’t rewrite my articles for each reader who has their own peculiar language preferences...
The real question is, do you agree with my characterization of the intended meaning of these intentionality-scented words (as used in particularly this article, say) as being mostly unrelated to normativity, that is to FAI-grade machine ethics? It is unclear to me if you agree or not. If there is some connection, what is it? It is also unclear to me how confusing or clear this question appears to other readers.
(On the other hand, who or what bears the blame for my (or others’) peculiar confusions is uninteresting.)
I don’t recall bringing up the issue of blame. All I’m saying is that I don’t have time to write a separate version of each post to accomodate each person’s language preferences, so I’m usually going to use the standard language used by the researchers in the field I’m discussing.
Words like ‘motivation’, ‘value’, ‘desire’, ‘want’ don’t have normative connotations in my head when I’m discussing them in the context of descriptivist neuroscience. The connotations in your brain may vary. I’m trying to discuss merely descriptive issues; I intend to start using descriptive facts to solve normative problems later. For now, I want to focus on getting a correct descriptive understanding of the system that causes humans do what they do before applying that knowledge to normative questions about what humans should do or what a Friendly AI should do.
Yes, it clarifies your intended meaning for the words, and resolves my confusion (for the second time; better watch for the confusing influence of those connotations in the future).
(I’m still deeply skeptical that descriptive understanding can help with FAI, but this is mostly unrelated to this post and others, which are good LW material when not confused for discussion of normativity.)
How would you (descriptively, “from the outside”) explain the fact that you didn’t provide information that would resolve my confusion (that you provided now), and instead pointed out that the reason for your actions lies in a tradition (conventional usage), and that I should engage that tradition directly? It seems like you were moved by considerations of assignment of blame (or responsibility), specifically you directed my attention to the process responsible for the problem. (I don’t expect you thought of this so explicitly, but still something caused your response to go the way it went.)
Blame is condemnation useful in shaping the future. It’s not latent in who had the best opportunity to avoid a problem, or the last clear chance to avoid a problem, or who began a problem, etc.
Responsibility is something political beings invent to relate agents to causation.
When people talk about causation they’re not necessarily playing that game.
Hmmm. I’m sorry you took it that way. I’m starting to get the sense that perhaps you see more connotations of normativity and judgment in general, and I try to see the world through the lens of a descriptivist project by default except for those rare occasions when I’m going to take a dangerous leap into the confusing lands of normativity.
How would you… explain the fact that you didn’t provide information that would resolve my confusion… and instead pointed out that the reason for your actions lies in a tradition...
I didn’t know which information would resolve your confusion until after I stumbled upon it. The point about common usage merely meant to explain why I’m using the terms the way I am.
(Strictly speaking, it’s not necessary to know something in order to be motivated by it. If a fact is considered relevant, but isn’t known, that creates instrumental motivation for finding out what it is! And even if you can’t learn something, you might want to establish a certain dependence of the outcome on that fact, no matter what the fact is.)
Sure, but what I’m saying is that this doesn’t happen in a way that allows your neurons to encode value for a 140-volt electrical shock.
Ah, understood.
But I’m not trying to show why encoding values for propositions is problematic. I’m trying to say that the brain does not encode values for objective intensities of stimuli.
Okay. I understand that it’s a fact that the brain doesn’t encode values for objective intensities of sensory stimuli. My puzzlement comes from when you say
Thus we smack headlong into another problem for human values and their extrapolation.
I don’t see the fact as an additional problem for a theory of human values. But there’s no point in arguing about this, as I think we’d both agree that any theory of human values would have to accommodate the fact.
Hmmm. Maybe a clearer way to say it is just that this neurobiological finding further constrains our theories. I’ll change the wording in the OP, thanks.
I’m not certain your examples of reference-dependent encoding of sense-data really demonstrate or have much to do with a lack of objective goals. (Of course, the framing effect example demonstrates this plenty well. :P ) As you point out, this is largely just adjusting for irrelevant background, like whether the sun is out, when what we care about has nothing to do with that. This is just throwing away the information at an early stage, rather than later after having explicitly determined that it’s irrelevant to our goals.
I agree that the framing effect is more important than the reference-dependence of sense-data encoding. However, the loss of sense-data is not always just “adjusting for irrelevant background”, and is not always throwing away something we would later have decided is “irrelevant to our goals.”
When I first read the post, I thought you were going to say something along the lines of:
“Evolution has optimized us to strip away the irrelevant features when it comes to vision, since it’s been vital for our survival. But evolution hasn’t done that for things like abstract value, since there’s been no selection pressure for that. It’s bad that our judgments in cases like the K&T examples don’t work more like vision, but that’s how it goes”.
Indeed, saying “let’s make the problem worse” and then bringing up vision feels a bit weird. After all, vision seems like a case where our brain does things exactly right—it ignores the “framing effects” caused by changed lightning conditions and leaves invariant the things that actually matter.
I wrote a response here.
An illuminating (no pun intended) example of when the adjustment to the ambient level of sense-data affects what people think they want would be nice. Without it the whole section seems to detract from your point.
I wrote a response here.
But I’m not raising a puzzle about how people think they want things even when they are behavioristic machines. I’m raising a puzzle about how we can be said to actually want things even when they are behavioristic machines that, for example, exhibit framing effects and can’t use neurons to encode value for the objective intensities of stimuli.
Suppose you have a neurological disorder that will be cured by a 140-volt electrical shock. If your brain can’t encode value for propositions or simulated states of affairs or anything like that, but only for stimuli, then this reference point business I described means that your brain doesn’t have the option of encoding value for a 140-volt electrical shock, because it never receives that kind of information in the first place. The transducer discards information about the objective intensity of the stimuli before the signal reaches the brain.
As Kaj says, this is a smart solution to lots of problems, but it does mean that the brain cannot encode value for the objective intensity of stimuli… at least given what I’ve explained in this post so far. (Model-based representations of value will be described later.)
Does that make sense?
If the neurological problem is located in the brain, then the brain does record information about the objective intensity of the stimuli, by being cured or not cured.
I’m confused about what the purpose of this example is. There are easier ways to show why not encoding values for propositions is problematic.
Sure, but what I’m saying is that this doesn’t happen in a way that allows your neurons to encode value for a 140-volt electrical shock. Perhaps you’ve already accepted this and find it obvious, but others (e.g. economists) do not. This kind of information about how the brain works constrains our models of human behavior, just like the stochasticity of neuron firing does.
But I’m not trying to show why encoding values for propositions is problematic. I’m trying to say that the brain does not encode values for objective intensities of stimuli.
Given that one could use propositions about objective intensities of stimuli (as you do now, to point out that what’s encoded in this particular simple way is not it), the thesis is still unclear.
Sure, but that depends on a different mechanism we don’t know much about, then. What I’m saying is that “Whaddyaknow, we discovered a mechanism that actually encodes value for stimuli with neuron firing rates! Ah, but it can’t encode value for the objective intensities of stimuli, because the brain doesn’t have that information. So that constrains our theories about the motivation of human behavior.”
The brain has (some measure of reference/access to) that information, just not in that particular form. And if it has (reference to) that information, it’s not possible to conclude that motivation doesn’t refer to it. It just doesn’t refer to it through exclusively the form of representation that doesn’t have the information, but then it would be very surprising if motivation compartmentalized so.
Right. I guess I’m struggling for a concise way to say what I’m trying to say, and hoping you’ll interpret me correctly based on the long paragraphs I’ve written explaining what I mean by these shorter sentences. Maybe something like:
“Whaddyaknow, we discovered a mechanism that actually encodes value for stimuli with neuron firing rates! Ah, but this particular mechanism can’t encode value for the objective intensities of stimuli, because this mechanism discards that information at the transducer. So that constrains our theories about the motivation of human behavior.”
Also, this doesn’t sound right. Why is that behavioral pattern “value”? Maybe it should be edited out of the system, like pain, or reversed, or modified in some complicated way.
Doesn’t really help. The problem is that (normative) motivation is the whole thing, particular (unusual) decisions can be formed by any component, so it’s unclear how to rule out stuff on the basis of properties of particular better-understood components.
Behavior is easier to analyze, you can see which factors contribute how much, and in this sense you can say that particular classes of behavior are determined mostly by this here mechanism that doesn’t have certain data, and so behavior is independent from that data. But such conclusions won’t generalize to normative motivation, because prevailing patterns of behavior might be suboptimal, and it’s possible to improve them (by exercising the less-prevalent modes of behavior that are less understood), making them depend on things that they presently don’t depend on.
What do you mean by ‘normative motivation’?
Considerations that should motivate you. What do you mean by “motivation”?
Uh oh. How did ‘should’ sneak its way into our discussion? I’m just talking about positive accounts of human motivation.
Until data give us a clearer picture of what we’re talking about, ‘motivation’ is whatever drives (apparently) goal-seeking behavior.
I guess the objection I have is to calling the behavioral summary “motivation”, a term that has normative connotations (similarly, “value”, “desire”, “wants”, etc.). Asking “Do we really want X?” (as in, does a positive account of some notion of “wanting” say that we “want” X, to the best of our scientific knowledge) sounds too similar to asking “Should we pursue X?” or even “Can we pursue X?”, but is a largely unrelated question with similarly unrelated answers.
I’m using these terms the way they are standardly used in the literature. If you object to the common usage, perhaps you could just read my articles with the assumption that I’m using these words the way neuroscientists and psychologists do, and then state your concerns about the standard language in the comments? I can’t rewrite my articles for each reader who has their own peculiar language preferences...
The real question is, do you agree with my characterization of the intended meaning of these intentionality-scented words (as used in particularly this article, say) as being mostly unrelated to normativity, that is to FAI-grade machine ethics? It is unclear to me if you agree or not. If there is some connection, what is it? It is also unclear to me how confusing or clear this question appears to other readers.
(On the other hand, who or what bears the blame for my (or others’) peculiar confusions is uninteresting.)
I don’t recall bringing up the issue of blame. All I’m saying is that I don’t have time to write a separate version of each post to accomodate each person’s language preferences, so I’m usually going to use the standard language used by the researchers in the field I’m discussing.
Words like ‘motivation’, ‘value’, ‘desire’, ‘want’ don’t have normative connotations in my head when I’m discussing them in the context of descriptivist neuroscience. The connotations in your brain may vary. I’m trying to discuss merely descriptive issues; I intend to start using descriptive facts to solve normative problems later. For now, I want to focus on getting a correct descriptive understanding of the system that causes humans do what they do before applying that knowledge to normative questions about what humans should do or what a Friendly AI should do.
Does that make sense?
Yes, it clarifies your intended meaning for the words, and resolves my confusion (for the second time; better watch for the confusing influence of those connotations in the future).
(I’m still deeply skeptical that descriptive understanding can help with FAI, but this is mostly unrelated to this post and others, which are good LW material when not confused for discussion of normativity.)
How would you (descriptively, “from the outside”) explain the fact that you didn’t provide information that would resolve my confusion (that you provided now), and instead pointed out that the reason for your actions lies in a tradition (conventional usage), and that I should engage that tradition directly? It seems like you were moved by considerations of assignment of blame (or responsibility), specifically you directed my attention to the process responsible for the problem. (I don’t expect you thought of this so explicitly, but still something caused your response to go the way it went.)
I don’t think blame works the way you seem to.
Blame is condemnation useful in shaping the future. It’s not latent in who had the best opportunity to avoid a problem, or the last clear chance to avoid a problem, or who began a problem, etc.
Responsibility is something political beings invent to relate agents to causation.
When people talk about causation they’re not necessarily playing that game.
Hmmm. I’m sorry you took it that way. I’m starting to get the sense that perhaps you see more connotations of normativity and judgment in general, and I try to see the world through the lens of a descriptivist project by default except for those rare occasions when I’m going to take a dangerous leap into the confusing lands of normativity.
I didn’t know which information would resolve your confusion until after I stumbled upon it. The point about common usage merely meant to explain why I’m using the terms the way I am.
(Strictly speaking, it’s not necessary to know something in order to be motivated by it. If a fact is considered relevant, but isn’t known, that creates instrumental motivation for finding out what it is! And even if you can’t learn something, you might want to establish a certain dependence of the outcome on that fact, no matter what the fact is.)
Ah, understood.
Okay. I understand that it’s a fact that the brain doesn’t encode values for objective intensities of sensory stimuli. My puzzlement comes from when you say
I don’t see the fact as an additional problem for a theory of human values. But there’s no point in arguing about this, as I think we’d both agree that any theory of human values would have to accommodate the fact.
Hmmm. Maybe a clearer way to say it is just that this neurobiological finding further constrains our theories. I’ll change the wording in the OP, thanks.