I can assign positive utility to whatever interpretation of an event I please. If the map changes, the utility changes, even if the territory stays the same. Preferences are not in the territory. Did I misunderstand you?
You haven’t misunderstood me, but you need to pay attention to this question, because it’s more or less a consensus on Less Wrong that your position expressed in the above quote is wrong. You should maybe ask around for clarification of this point, if you don’t get a change of mind from discussion with me.
You may try the metaethics sequence, and also/in particular these posts:
That preference is computed in the mind doesn’t make it any less of territory than anything else. This is just a piece of territory that happens to be currently located in human minds. (Well, not quite, but to a first approximation.)
Your map may easily change even if the territory stays the same. This changes your belief, but this change doesn’t influence what’s true about the territory. Likewise, your estimate of how good situation X is may change, once you process new arguments or change your understanding of the situation, for example by observing new data, but that change of your belief doesn’t influence how good X actually is. Morality is not a matter of interpretation.
Before I spend a lot of effort trying to figure out where I went wrong (which I’m completely willing to do, because I read all of those posts and the metaethics sequence and figured I understood them), can you confirm that you read my EDIT above, and that the misunderstanding addressed there does not encompass the problem?
Now I have read the edit, but it doesn’t seem to address the problem. Also, I don’t see what you can use the concepts you bring up for, like “probability that I will get enough utility to justify cryonics upon reflection”. If you expect to believe something, you should just believe it right away. See Conservation of expected evidence. But then, “probability this decision is right” is not something you can use for making the decision, not directly.
Also, I don’t see what you can use the concepts you bring up for, like “probability that I will get enough utility to justify cryonics upon reflection”.
This might not be the most useful concept, true, but the issue at hand is the meta-level one of people’s possible overconfidence about it.
“Probability of signing up being good”, especially obfuscated with “justified upon infinite reflection”, being subtly similar to “probability of the decision to sign up being correct”, is too much of a ruse to use without very careful elaboration. A decision can be absolutely, 99.999999% correct, while the probability of it being good remains at 1%, both known to the decider.
So you read footnote 2 of the post and do not think it is a relevant and necessary distinction? And you read Steven’s comment in the other thread where it seems he dissolved our disagreement and determined we were talking about different things?
I know about the conservation of expected evidence. I understand and have demonstrated understanding of the content in the various links you’ve given me. I really doubt I’ve been making the obvious errors you accuse me of for the many months I’ve been conversing with people at SIAI (and at Less Wrong meetups and at the decision theory workshop) without anyone noticing.
Here’s a basic summary of what you seem to think I’m confused about: There is a broad concept of identity in my head. Given this concept of identity I do not want to sign up for cryonics. If this concept of identity changed such that the set of computations I identified with became smaller, then cryonics would become more appealing. I am talking about the probability of expected utility, not the probability of an event. The first is in the map (even if the map is in the territory, which I realize, of course), the second is in the territory.
EDIT: I am treating considerations about identity as a preference: whether or not I should identify with any set of computations is my choice, but subject to change. I think that might be where we disagree: you think everybody will eventually agree what identity is, and that it will be considered a fact about which we can assign different probabilities, but not something subjectively determined.
I am treating considerations about identity as a preference: whether or not I should identify with any set of computations is my choice, but subject to change. I think that might be where we disagree: you think everybody will eventually agree what identity is, and that it will be considered a fact about which we can assign different probabilities, but not something subjectively determined.
That preference is yours and yours alone, without any community to share it, doesn’t make its content any less of a fact than if you’d had a whole humanity of identical people to back it up. (This identity/probability discussion is tangential to a more focused question of correctness of choice.)
The easiest step is for you to look over the last two paragraphs of this comment and see if you agree with that. (Agree/disagree in what sense, if you suspect essential interpretational ambiguity.)
I don’t know why you brought up the concept of identity (or indeed cryonics) in the above, it wasn’t part of this particular discussion.
At first glance and 15 seconds of thinking, I agree, but: “but that change of your belief doesn’t influence how good X actually is” is to me more like “but that change of your belief doesn’t influence how good X will be considered upon an infinite amount of infinitely good reflection”.
Now try to figure out what does the question “What color the sky actually is?” mean, when compared with “How good X actually is?” and your interpretation “How good will X seem after infinite amount of infinitely good reflection”. The “infinitely good reflection” thing is a surrogate for the fact itself, no less in the first case, and no more in the second.
If you essentially agree that there is fact of the matter about whether a given decision is the right one, what did you mean by the following?
I can assign positive utility to whatever interpretation of an event I please. If the map changes, the utility changes, even if the territory stays the same. Preferences are not in the territory.
You can’t “assign utility as you please”, this is not a matter of choice. The decision is either correct or it isn’t, and you can’t make it correct or incorrect by willing so. You may only work on figuring out which way it is, like with any other fact.
Edit: adding a sentence in bold that is really important but that I failed to notice the first time. (Nick Tarleton alerted me to an error in this comment that I needed to fix.)
Any intelligent agent will discover that the sky is blue. Not every intelligent agent will think that the blue sky is equally beautiful. Me, I like grey skies and rainy days. If I discover that I actually like blue skies at a later point, then that changes the perceived utility of seeing a grey sky relative to a blue one. The simple change in preference also changes my expected utility. Yes, maybe the new utility was the ‘correct’ utility all along, but how is that an argument against anything I’ve said in my posts or comments? I get the impression you consistently take the territory view where I take the map view, and I further think that the map view is way more useful for agents like me that aren’t infinitely intelligent nor infinitely reflective. (Nick Tarleton disagrees about taking the map view and I am now reconsidering. He raises the important point that taking the territory view doesn’t mean throwing out the map, and gives the map something to be about. I think he’s probably right.)
You may only work on figuring out which way it is, like with any other fact.
And the way one does this is by becoming good at luminosity and discovering what one’s terminal values are. Yeah, maybe it turns out sufficiently intelligent agents all end up valuing the exact same thing, and FAI turns out to be really easy, but I do not buy it as an assertion.
And the way one does this is by becoming good at luminosity and discovering what one’s terminal values are. Yeah, maybe it turns out sufficiently intelligent agents all end up valuing the exact same thing, and FAI turns out to be really easy, but I do not buy it as an assertion.
This reads to me like
To figure out the weight of a person, we need to develop experimental procedures, make observations, and so on. Yes, maybe it turns out that “weight of a person” is a universal constant and that all experimenters will agree that it’s exactly 80 kg in all cases, and weighting people will thus turn out really easy, but I don’t buy this assertion.
See the error? That there are moral facts doesn’t imply that everyone’s preference is identical, that “all intelligent agents” will value the same thing. Every sane agent should agree on what is moral, but not every sane agent is moved by what is moral, some may be moved by what is prime or something, while agreeing with you that what is prime is often not moral. (See also this comment.)
I’m a little confused about your “weight of a person” example because ‘a’ is ambiguous in English. Did you mean one specific person, or the weighing of different people?
Every sane agent should agree on what is moral
What if CEV doesn’t exist, and there really are different groups of humans with different values? Is one set of values “moral” and the other “that other human thing that’s analogous to morality but isn’t morality”? Primeness is so different from morality that it’s clear we’re talking about two different things. But say we take what you’re calling morality and modify it very slightly, only to the point where many humans still hold to the modified view. It’s not clear to me that the agents will say “I’m moved by this modified view, not morality”. Why wouldn’t they say “No, this modification is the correct morality, and I am moved by morality!”
I have read the metaethics sequence but don’t claim to fully understand it, so feel free to point me to a particular part of it.
What if CEV doesn’t exist, and there really are different groups of humans with different values?
Of course different people have different values. These values might be similar, but they won’t be identical.
Primeness is so different from morality that it’s clear we’re talking about two different things.
Yes, but what is “prime number”? Is it 5, or is it 7? 5 is clearly different from 7, although it’s very similar to it in that it’s also prime. Use the analogy of prime=moral and 5=Blueberry’s values, 7=Will’s values.
It’s not clear to me that the agents will say “I’m moved by this modified view, not morality”. Why wouldn’t they say “No, this modification is the correct morality, and I am moved by morality!”
Because that would be pointless disputing of definitions—clearly, different things are meant by word “morality” in your example.
Yes, but what is “prime number”? Is it 5, or is it 7? 5 is clearly different from 7, although it’s very similar to it in that it’s also prime. Use the analogy of prime=moral and 5=Blueberry’s values, 7=Will’s values.
I see your point, but there is an obvious problem with this analogy: prime and nonprime are two discrete categories. But we can consider a continuum of values, ranging from something almost everyone agrees is moral, through values that are unusual or uncommon but still recognized as human values, all the way to completely alien values like paperclipping.
My concern is that it’s not clear where in the continuum the values stop being “moral” values, unlike with prime numbers.
My concern is that it’s not clear where in the continuum the values stop being “moral” values, unlike with prime numbers.
It might be unclear where the line lies, but it shouldn’t make the concept itself “fuzzy”, merely not understood. What we talk about when we refer to a certain idea is always something specific, but it’s not always clear what is implied by what we talk about. That different people can interpret the same words as referring to different ideas doesn’t make any of these different ideas undefined. The failure to interpret the words in the same way is a failure of communication, not a characterization of the idea that failed to be communicated.
I of course agree that “morality” admits a lot of similar interpretations, but I’d venture to say that “Blueberry’s preference” does as well. It’s an unsolved problem—a core question of Friendly AI—to formally define any of the concepts interpreting these words in a satisfactory way. The fuzziness in communication and elusiveness in formal understanding are relevant equally for the aggregate morality and personal preference, and so the individual/aggregate divide is not the point that particularly opposes the analogy.
Do you think there is a clear line between what humans in general value (morality) and what other entities might value, and we just don’t know where it is? Let’s call the other side of the line ‘schmorality’. So a paperclipper’s values are schmoral.
Is it possible that a human could have values on the other side of the line (schmoral values)?
Suppose another entity, who is on the other side of the line, has a conversation with a human about a moral issue. Both entities engage in the same kind of reasoning, use the same kind of arguments and examples, so why is one reasoning called “moral reasoning” and the other just about values (schmoral reasoning)?
Suppose I am right on the edge of the line. So my values are moral values, but a slight change makes these values schmoral values. From my point of view, these two sets of values are very close. Why do you give them completely different categories? And suppose my values change slightly over time, so I cross the line and back within a day. Do I suddenly stop caring about morality, then start again? This discontinuity seems very strange to me.
I don’t say that any given concept is reasonable for all purposes, just that any concept has a very specific intended meaning, at the moment it’s considered. The concept of morality can be characterized as, roughly, referring to human-like preference, or aggregate preference of humanity-like collections of individual preferences—this is a characterization resilient to some measure of ambiguity in interpretation. The concepts themselves can’t be negotiated, they are set in stone by their intended meaning, though a different concept may be better for a given purpose.
I don’t say that any given concept is reasonable for all purposes, just that any concept has a very specific intended meaning, at the moment it’s considered. The concept of morality can be characterized as, roughly, referring to human-like preference
If you essentially agree that there is fact of the matter about whether a given decision is the right one, what did you mean by the following?
I can assign positive utility to whatever interpretation of an event I please. If the map changes, the utility changes, even if the territory stays the same. Preferences are not in the territory.
In this exchange
If Will’s probability is correct, then I fail to see how his post makes sense: it wouldn’t make sense for anyone to pay for cryo.
There are important subjective considerations, such as age and definition of identity,
Nope, “definition of identity” doesn’t influence what actually happens as a result of your decision, and thus doesn’t influence how good what happens will be.
Will, by “definition of identity”, meant a part of preference, making the point that people might have varying preferences (this being the sense in which preference is “subjective”) that make cryonics a good idea for some but not others. He read your response as a statement of something like moral realism/externalism; he intended his response to address this, though it was phrased confusingly.
That would be a potentially defensible view (What are the causes of variation? How do we know it’s there?), but I’m not sure it’s Will’s (and using the word “definition” in this sense goes very much against the definition of “definition”).
You haven’t misunderstood me, but you need to pay attention to this question, because it’s more or less a consensus on Less Wrong that your position expressed in the above quote is wrong. You should maybe ask around for clarification of this point, if you don’t get a change of mind from discussion with me.
You may try the metaethics sequence, and also/in particular these posts:
http://lesswrong.com/lw/s6/probability_is_subjectively_objective/
http://lesswrong.com/lw/si/math_is_subjunctively_objective/
http://lesswrong.com/lw/sj/does_your_morality_care_what_you_think/
http://lesswrong.com/lw/sw/morality_as_fixed_computation/
http://lesswrong.com/lw/t0/abstracted_idealized_dynamics/
That preference is computed in the mind doesn’t make it any less of territory than anything else. This is just a piece of territory that happens to be currently located in human minds. (Well, not quite, but to a first approximation.)
Your map may easily change even if the territory stays the same. This changes your belief, but this change doesn’t influence what’s true about the territory. Likewise, your estimate of how good situation X is may change, once you process new arguments or change your understanding of the situation, for example by observing new data, but that change of your belief doesn’t influence how good X actually is. Morality is not a matter of interpretation.
Before I spend a lot of effort trying to figure out where I went wrong (which I’m completely willing to do, because I read all of those posts and the metaethics sequence and figured I understood them), can you confirm that you read my EDIT above, and that the misunderstanding addressed there does not encompass the problem?
Now I have read the edit, but it doesn’t seem to address the problem. Also, I don’t see what you can use the concepts you bring up for, like “probability that I will get enough utility to justify cryonics upon reflection”. If you expect to believe something, you should just believe it right away. See Conservation of expected evidence. But then, “probability this decision is right” is not something you can use for making the decision, not directly.
This might not be the most useful concept, true, but the issue at hand is the meta-level one of people’s possible overconfidence about it.
“Probability of signing up being good”, especially obfuscated with “justified upon infinite reflection”, being subtly similar to “probability of the decision to sign up being correct”, is too much of a ruse to use without very careful elaboration. A decision can be absolutely, 99.999999% correct, while the probability of it being good remains at 1%, both known to the decider.
So you read footnote 2 of the post and do not think it is a relevant and necessary distinction? And you read Steven’s comment in the other thread where it seems he dissolved our disagreement and determined we were talking about different things?
I know about the conservation of expected evidence. I understand and have demonstrated understanding of the content in the various links you’ve given me. I really doubt I’ve been making the obvious errors you accuse me of for the many months I’ve been conversing with people at SIAI (and at Less Wrong meetups and at the decision theory workshop) without anyone noticing.
Here’s a basic summary of what you seem to think I’m confused about: There is a broad concept of identity in my head. Given this concept of identity I do not want to sign up for cryonics. If this concept of identity changed such that the set of computations I identified with became smaller, then cryonics would become more appealing. I am talking about the probability of expected utility, not the probability of an event. The first is in the map (even if the map is in the territory, which I realize, of course), the second is in the territory.
EDIT: I am treating considerations about identity as a preference: whether or not I should identify with any set of computations is my choice, but subject to change. I think that might be where we disagree: you think everybody will eventually agree what identity is, and that it will be considered a fact about which we can assign different probabilities, but not something subjectively determined.
That preference is yours and yours alone, without any community to share it, doesn’t make its content any less of a fact than if you’d had a whole humanity of identical people to back it up. (This identity/probability discussion is tangential to a more focused question of correctness of choice.)
The easiest step is for you to look over the last two paragraphs of this comment and see if you agree with that. (Agree/disagree in what sense, if you suspect essential interpretational ambiguity.)
I don’t know why you brought up the concept of identity (or indeed cryonics) in the above, it wasn’t part of this particular discussion.
At first glance and 15 seconds of thinking, I agree, but: “but that change of your belief doesn’t influence how good X actually is” is to me more like “but that change of your belief doesn’t influence how good X will be considered upon an infinite amount of infinitely good reflection”.
Now try to figure out what does the question “What color the sky actually is?” mean, when compared with “How good X actually is?” and your interpretation “How good will X seem after infinite amount of infinitely good reflection”. The “infinitely good reflection” thing is a surrogate for the fact itself, no less in the first case, and no more in the second.
If you essentially agree that there is fact of the matter about whether a given decision is the right one, what did you mean by the following?
You can’t “assign utility as you please”, this is not a matter of choice. The decision is either correct or it isn’t, and you can’t make it correct or incorrect by willing so. You may only work on figuring out which way it is, like with any other fact.
Edit: adding a sentence in bold that is really important but that I failed to notice the first time. (Nick Tarleton alerted me to an error in this comment that I needed to fix.)
Any intelligent agent will discover that the sky is blue. Not every intelligent agent will think that the blue sky is equally beautiful. Me, I like grey skies and rainy days. If I discover that I actually like blue skies at a later point, then that changes the perceived utility of seeing a grey sky relative to a blue one. The simple change in preference also changes my expected utility. Yes, maybe the new utility was the ‘correct’ utility all along, but how is that an argument against anything I’ve said in my posts or comments? I get the impression you consistently take the territory view where I take the map view, and I further think that the map view is way more useful for agents like me that aren’t infinitely intelligent nor infinitely reflective. (Nick Tarleton disagrees about taking the map view and I am now reconsidering. He raises the important point that taking the territory view doesn’t mean throwing out the map, and gives the map something to be about. I think he’s probably right.)
And the way one does this is by becoming good at luminosity and discovering what one’s terminal values are. Yeah, maybe it turns out sufficiently intelligent agents all end up valuing the exact same thing, and FAI turns out to be really easy, but I do not buy it as an assertion.
This reads to me like
See the error? That there are moral facts doesn’t imply that everyone’s preference is identical, that “all intelligent agents” will value the same thing. Every sane agent should agree on what is moral, but not every sane agent is moved by what is moral, some may be moved by what is prime or something, while agreeing with you that what is prime is often not moral. (See also this comment.)
I’m a little confused about your “weight of a person” example because ‘a’ is ambiguous in English. Did you mean one specific person, or the weighing of different people?
What if CEV doesn’t exist, and there really are different groups of humans with different values? Is one set of values “moral” and the other “that other human thing that’s analogous to morality but isn’t morality”? Primeness is so different from morality that it’s clear we’re talking about two different things. But say we take what you’re calling morality and modify it very slightly, only to the point where many humans still hold to the modified view. It’s not clear to me that the agents will say “I’m moved by this modified view, not morality”. Why wouldn’t they say “No, this modification is the correct morality, and I am moved by morality!”
I have read the metaethics sequence but don’t claim to fully understand it, so feel free to point me to a particular part of it.
Of course different people have different values. These values might be similar, but they won’t be identical.
Yes, but what is “prime number”? Is it 5, or is it 7? 5 is clearly different from 7, although it’s very similar to it in that it’s also prime. Use the analogy of prime=moral and 5=Blueberry’s values, 7=Will’s values.
Because that would be pointless disputing of definitions—clearly, different things are meant by word “morality” in your example.
I see your point, but there is an obvious problem with this analogy: prime and nonprime are two discrete categories. But we can consider a continuum of values, ranging from something almost everyone agrees is moral, through values that are unusual or uncommon but still recognized as human values, all the way to completely alien values like paperclipping.
My concern is that it’s not clear where in the continuum the values stop being “moral” values, unlike with prime numbers.
It might be unclear where the line lies, but it shouldn’t make the concept itself “fuzzy”, merely not understood. What we talk about when we refer to a certain idea is always something specific, but it’s not always clear what is implied by what we talk about. That different people can interpret the same words as referring to different ideas doesn’t make any of these different ideas undefined. The failure to interpret the words in the same way is a failure of communication, not a characterization of the idea that failed to be communicated.
I of course agree that “morality” admits a lot of similar interpretations, but I’d venture to say that “Blueberry’s preference” does as well. It’s an unsolved problem—a core question of Friendly AI—to formally define any of the concepts interpreting these words in a satisfactory way. The fuzziness in communication and elusiveness in formal understanding are relevant equally for the aggregate morality and personal preference, and so the individual/aggregate divide is not the point that particularly opposes the analogy.
I’m still very confused.
Do you think there is a clear line between what humans in general value (morality) and what other entities might value, and we just don’t know where it is? Let’s call the other side of the line ‘schmorality’. So a paperclipper’s values are schmoral.
Is it possible that a human could have values on the other side of the line (schmoral values)?
Suppose another entity, who is on the other side of the line, has a conversation with a human about a moral issue. Both entities engage in the same kind of reasoning, use the same kind of arguments and examples, so why is one reasoning called “moral reasoning” and the other just about values (schmoral reasoning)?
Suppose I am right on the edge of the line. So my values are moral values, but a slight change makes these values schmoral values. From my point of view, these two sets of values are very close. Why do you give them completely different categories? And suppose my values change slightly over time, so I cross the line and back within a day. Do I suddenly stop caring about morality, then start again? This discontinuity seems very strange to me.
I don’t say that any given concept is reasonable for all purposes, just that any concept has a very specific intended meaning, at the moment it’s considered. The concept of morality can be characterized as, roughly, referring to human-like preference, or aggregate preference of humanity-like collections of individual preferences—this is a characterization resilient to some measure of ambiguity in interpretation. The concepts themselves can’t be negotiated, they are set in stone by their intended meaning, though a different concept may be better for a given purpose.
Thanks! That actually helped a lot.
In this exchange
Will, by “definition of identity”, meant a part of preference, making the point that people might have varying preferences (this being the sense in which preference is “subjective”) that make cryonics a good idea for some but not others. He read your response as a statement of something like moral realism/externalism; he intended his response to address this, though it was phrased confusingly.
That would be a potentially defensible view (What are the causes of variation? How do we know it’s there?), but I’m not sure it’s Will’s (and using the word “definition” in this sense goes very much against the definition of “definition”).