Now try to figure out what does the question “What color the sky actually is?” mean, when compared with “How good X actually is?” and your interpretation “How good will X seem after infinite amount of infinitely good reflection”. The “infinitely good reflection” thing is a surrogate for the fact itself, no less in the first case, and no more in the second.
If you essentially agree that there is fact of the matter about whether a given decision is the right one, what did you mean by the following?
I can assign positive utility to whatever interpretation of an event I please. If the map changes, the utility changes, even if the territory stays the same. Preferences are not in the territory.
You can’t “assign utility as you please”, this is not a matter of choice. The decision is either correct or it isn’t, and you can’t make it correct or incorrect by willing so. You may only work on figuring out which way it is, like with any other fact.
Edit: adding a sentence in bold that is really important but that I failed to notice the first time. (Nick Tarleton alerted me to an error in this comment that I needed to fix.)
Any intelligent agent will discover that the sky is blue. Not every intelligent agent will think that the blue sky is equally beautiful. Me, I like grey skies and rainy days. If I discover that I actually like blue skies at a later point, then that changes the perceived utility of seeing a grey sky relative to a blue one. The simple change in preference also changes my expected utility. Yes, maybe the new utility was the ‘correct’ utility all along, but how is that an argument against anything I’ve said in my posts or comments? I get the impression you consistently take the territory view where I take the map view, and I further think that the map view is way more useful for agents like me that aren’t infinitely intelligent nor infinitely reflective. (Nick Tarleton disagrees about taking the map view and I am now reconsidering. He raises the important point that taking the territory view doesn’t mean throwing out the map, and gives the map something to be about. I think he’s probably right.)
You may only work on figuring out which way it is, like with any other fact.
And the way one does this is by becoming good at luminosity and discovering what one’s terminal values are. Yeah, maybe it turns out sufficiently intelligent agents all end up valuing the exact same thing, and FAI turns out to be really easy, but I do not buy it as an assertion.
And the way one does this is by becoming good at luminosity and discovering what one’s terminal values are. Yeah, maybe it turns out sufficiently intelligent agents all end up valuing the exact same thing, and FAI turns out to be really easy, but I do not buy it as an assertion.
This reads to me like
To figure out the weight of a person, we need to develop experimental procedures, make observations, and so on. Yes, maybe it turns out that “weight of a person” is a universal constant and that all experimenters will agree that it’s exactly 80 kg in all cases, and weighting people will thus turn out really easy, but I don’t buy this assertion.
See the error? That there are moral facts doesn’t imply that everyone’s preference is identical, that “all intelligent agents” will value the same thing. Every sane agent should agree on what is moral, but not every sane agent is moved by what is moral, some may be moved by what is prime or something, while agreeing with you that what is prime is often not moral. (See also this comment.)
I’m a little confused about your “weight of a person” example because ‘a’ is ambiguous in English. Did you mean one specific person, or the weighing of different people?
Every sane agent should agree on what is moral
What if CEV doesn’t exist, and there really are different groups of humans with different values? Is one set of values “moral” and the other “that other human thing that’s analogous to morality but isn’t morality”? Primeness is so different from morality that it’s clear we’re talking about two different things. But say we take what you’re calling morality and modify it very slightly, only to the point where many humans still hold to the modified view. It’s not clear to me that the agents will say “I’m moved by this modified view, not morality”. Why wouldn’t they say “No, this modification is the correct morality, and I am moved by morality!”
I have read the metaethics sequence but don’t claim to fully understand it, so feel free to point me to a particular part of it.
What if CEV doesn’t exist, and there really are different groups of humans with different values?
Of course different people have different values. These values might be similar, but they won’t be identical.
Primeness is so different from morality that it’s clear we’re talking about two different things.
Yes, but what is “prime number”? Is it 5, or is it 7? 5 is clearly different from 7, although it’s very similar to it in that it’s also prime. Use the analogy of prime=moral and 5=Blueberry’s values, 7=Will’s values.
It’s not clear to me that the agents will say “I’m moved by this modified view, not morality”. Why wouldn’t they say “No, this modification is the correct morality, and I am moved by morality!”
Because that would be pointless disputing of definitions—clearly, different things are meant by word “morality” in your example.
Yes, but what is “prime number”? Is it 5, or is it 7? 5 is clearly different from 7, although it’s very similar to it in that it’s also prime. Use the analogy of prime=moral and 5=Blueberry’s values, 7=Will’s values.
I see your point, but there is an obvious problem with this analogy: prime and nonprime are two discrete categories. But we can consider a continuum of values, ranging from something almost everyone agrees is moral, through values that are unusual or uncommon but still recognized as human values, all the way to completely alien values like paperclipping.
My concern is that it’s not clear where in the continuum the values stop being “moral” values, unlike with prime numbers.
My concern is that it’s not clear where in the continuum the values stop being “moral” values, unlike with prime numbers.
It might be unclear where the line lies, but it shouldn’t make the concept itself “fuzzy”, merely not understood. What we talk about when we refer to a certain idea is always something specific, but it’s not always clear what is implied by what we talk about. That different people can interpret the same words as referring to different ideas doesn’t make any of these different ideas undefined. The failure to interpret the words in the same way is a failure of communication, not a characterization of the idea that failed to be communicated.
I of course agree that “morality” admits a lot of similar interpretations, but I’d venture to say that “Blueberry’s preference” does as well. It’s an unsolved problem—a core question of Friendly AI—to formally define any of the concepts interpreting these words in a satisfactory way. The fuzziness in communication and elusiveness in formal understanding are relevant equally for the aggregate morality and personal preference, and so the individual/aggregate divide is not the point that particularly opposes the analogy.
Do you think there is a clear line between what humans in general value (morality) and what other entities might value, and we just don’t know where it is? Let’s call the other side of the line ‘schmorality’. So a paperclipper’s values are schmoral.
Is it possible that a human could have values on the other side of the line (schmoral values)?
Suppose another entity, who is on the other side of the line, has a conversation with a human about a moral issue. Both entities engage in the same kind of reasoning, use the same kind of arguments and examples, so why is one reasoning called “moral reasoning” and the other just about values (schmoral reasoning)?
Suppose I am right on the edge of the line. So my values are moral values, but a slight change makes these values schmoral values. From my point of view, these two sets of values are very close. Why do you give them completely different categories? And suppose my values change slightly over time, so I cross the line and back within a day. Do I suddenly stop caring about morality, then start again? This discontinuity seems very strange to me.
I don’t say that any given concept is reasonable for all purposes, just that any concept has a very specific intended meaning, at the moment it’s considered. The concept of morality can be characterized as, roughly, referring to human-like preference, or aggregate preference of humanity-like collections of individual preferences—this is a characterization resilient to some measure of ambiguity in interpretation. The concepts themselves can’t be negotiated, they are set in stone by their intended meaning, though a different concept may be better for a given purpose.
I don’t say that any given concept is reasonable for all purposes, just that any concept has a very specific intended meaning, at the moment it’s considered. The concept of morality can be characterized as, roughly, referring to human-like preference
If you essentially agree that there is fact of the matter about whether a given decision is the right one, what did you mean by the following?
I can assign positive utility to whatever interpretation of an event I please. If the map changes, the utility changes, even if the territory stays the same. Preferences are not in the territory.
In this exchange
If Will’s probability is correct, then I fail to see how his post makes sense: it wouldn’t make sense for anyone to pay for cryo.
There are important subjective considerations, such as age and definition of identity,
Nope, “definition of identity” doesn’t influence what actually happens as a result of your decision, and thus doesn’t influence how good what happens will be.
Will, by “definition of identity”, meant a part of preference, making the point that people might have varying preferences (this being the sense in which preference is “subjective”) that make cryonics a good idea for some but not others. He read your response as a statement of something like moral realism/externalism; he intended his response to address this, though it was phrased confusingly.
That would be a potentially defensible view (What are the causes of variation? How do we know it’s there?), but I’m not sure it’s Will’s (and using the word “definition” in this sense goes very much against the definition of “definition”).
Now try to figure out what does the question “What color the sky actually is?” mean, when compared with “How good X actually is?” and your interpretation “How good will X seem after infinite amount of infinitely good reflection”. The “infinitely good reflection” thing is a surrogate for the fact itself, no less in the first case, and no more in the second.
If you essentially agree that there is fact of the matter about whether a given decision is the right one, what did you mean by the following?
You can’t “assign utility as you please”, this is not a matter of choice. The decision is either correct or it isn’t, and you can’t make it correct or incorrect by willing so. You may only work on figuring out which way it is, like with any other fact.
Edit: adding a sentence in bold that is really important but that I failed to notice the first time. (Nick Tarleton alerted me to an error in this comment that I needed to fix.)
Any intelligent agent will discover that the sky is blue. Not every intelligent agent will think that the blue sky is equally beautiful. Me, I like grey skies and rainy days. If I discover that I actually like blue skies at a later point, then that changes the perceived utility of seeing a grey sky relative to a blue one. The simple change in preference also changes my expected utility. Yes, maybe the new utility was the ‘correct’ utility all along, but how is that an argument against anything I’ve said in my posts or comments? I get the impression you consistently take the territory view where I take the map view, and I further think that the map view is way more useful for agents like me that aren’t infinitely intelligent nor infinitely reflective. (Nick Tarleton disagrees about taking the map view and I am now reconsidering. He raises the important point that taking the territory view doesn’t mean throwing out the map, and gives the map something to be about. I think he’s probably right.)
And the way one does this is by becoming good at luminosity and discovering what one’s terminal values are. Yeah, maybe it turns out sufficiently intelligent agents all end up valuing the exact same thing, and FAI turns out to be really easy, but I do not buy it as an assertion.
This reads to me like
See the error? That there are moral facts doesn’t imply that everyone’s preference is identical, that “all intelligent agents” will value the same thing. Every sane agent should agree on what is moral, but not every sane agent is moved by what is moral, some may be moved by what is prime or something, while agreeing with you that what is prime is often not moral. (See also this comment.)
I’m a little confused about your “weight of a person” example because ‘a’ is ambiguous in English. Did you mean one specific person, or the weighing of different people?
What if CEV doesn’t exist, and there really are different groups of humans with different values? Is one set of values “moral” and the other “that other human thing that’s analogous to morality but isn’t morality”? Primeness is so different from morality that it’s clear we’re talking about two different things. But say we take what you’re calling morality and modify it very slightly, only to the point where many humans still hold to the modified view. It’s not clear to me that the agents will say “I’m moved by this modified view, not morality”. Why wouldn’t they say “No, this modification is the correct morality, and I am moved by morality!”
I have read the metaethics sequence but don’t claim to fully understand it, so feel free to point me to a particular part of it.
Of course different people have different values. These values might be similar, but they won’t be identical.
Yes, but what is “prime number”? Is it 5, or is it 7? 5 is clearly different from 7, although it’s very similar to it in that it’s also prime. Use the analogy of prime=moral and 5=Blueberry’s values, 7=Will’s values.
Because that would be pointless disputing of definitions—clearly, different things are meant by word “morality” in your example.
I see your point, but there is an obvious problem with this analogy: prime and nonprime are two discrete categories. But we can consider a continuum of values, ranging from something almost everyone agrees is moral, through values that are unusual or uncommon but still recognized as human values, all the way to completely alien values like paperclipping.
My concern is that it’s not clear where in the continuum the values stop being “moral” values, unlike with prime numbers.
It might be unclear where the line lies, but it shouldn’t make the concept itself “fuzzy”, merely not understood. What we talk about when we refer to a certain idea is always something specific, but it’s not always clear what is implied by what we talk about. That different people can interpret the same words as referring to different ideas doesn’t make any of these different ideas undefined. The failure to interpret the words in the same way is a failure of communication, not a characterization of the idea that failed to be communicated.
I of course agree that “morality” admits a lot of similar interpretations, but I’d venture to say that “Blueberry’s preference” does as well. It’s an unsolved problem—a core question of Friendly AI—to formally define any of the concepts interpreting these words in a satisfactory way. The fuzziness in communication and elusiveness in formal understanding are relevant equally for the aggregate morality and personal preference, and so the individual/aggregate divide is not the point that particularly opposes the analogy.
I’m still very confused.
Do you think there is a clear line between what humans in general value (morality) and what other entities might value, and we just don’t know where it is? Let’s call the other side of the line ‘schmorality’. So a paperclipper’s values are schmoral.
Is it possible that a human could have values on the other side of the line (schmoral values)?
Suppose another entity, who is on the other side of the line, has a conversation with a human about a moral issue. Both entities engage in the same kind of reasoning, use the same kind of arguments and examples, so why is one reasoning called “moral reasoning” and the other just about values (schmoral reasoning)?
Suppose I am right on the edge of the line. So my values are moral values, but a slight change makes these values schmoral values. From my point of view, these two sets of values are very close. Why do you give them completely different categories? And suppose my values change slightly over time, so I cross the line and back within a day. Do I suddenly stop caring about morality, then start again? This discontinuity seems very strange to me.
I don’t say that any given concept is reasonable for all purposes, just that any concept has a very specific intended meaning, at the moment it’s considered. The concept of morality can be characterized as, roughly, referring to human-like preference, or aggregate preference of humanity-like collections of individual preferences—this is a characterization resilient to some measure of ambiguity in interpretation. The concepts themselves can’t be negotiated, they are set in stone by their intended meaning, though a different concept may be better for a given purpose.
Thanks! That actually helped a lot.
In this exchange
Will, by “definition of identity”, meant a part of preference, making the point that people might have varying preferences (this being the sense in which preference is “subjective”) that make cryonics a good idea for some but not others. He read your response as a statement of something like moral realism/externalism; he intended his response to address this, though it was phrased confusingly.
That would be a potentially defensible view (What are the causes of variation? How do we know it’s there?), but I’m not sure it’s Will’s (and using the word “definition” in this sense goes very much against the definition of “definition”).