And if it turns out that humans currently want something different than what we wanted a thousand years ago, then it follows that a thousand years ago we didn’t want what was right, and now we do… though if you’d asked us a thousand years ago, we’d have said that we want what is right, and we’d have arrived at that conclusion through exactly the same cognitive operations we’re currently using. (Of course, in that case we would be mistaken, unlike the current case.)
And if it turns out that a thousand years from now humans want something different, then we will no longer want what is right… though if you ask us then, we’ll say we want what is right, again using the same cognitive operations. (Again, in that case we would be mistaken.)
And if there turn out to be two groups of humans who want incompatible things (for example, because their brains are sufficiently different), then whichever group I happen to be in wants what is right, and the other group doesn’t… though if you ask them, they’ll (mistakenly) say they want what is right, again using the same cognitive operations.
All of which strikes me as a pointlessly confusing way of saying that I endorse what humans-sufficiently-like-me currently want, and don’t endorse what we used to want or come to want or what anyone else wants if it’s too different from that.
Talking about whether some action is right or wrong or moral seems altogether unnecessary on this view. It is enough to say that I endorse what I value, and will program FAI to optimize for that, and will reject moral arguments that are inconsistent with that, and etc. Sure, if I valued something different, I would endorse that instead, but that doesn’t change anything; if I were hit by a speeding train, I’d be dead, but it doesn’t follow that I am dead. I endorse what I value, which means I consider worlds in which there is less of what I value worse than worlds in which there is more of what I value—even if those worlds also include versions of me that endorse something different. Fine and dandy.
What is added to that description by introducing words like right and wrong and moral, other than the confusion caused by people who assume those words refer to a magic light from the sky? It seems no more useful, on this view, than talking about how certain acts are salvatory or diabolical or fleabag.
If the people a thousand years ago might have wanted what is right, but were mistaken as to what they really wanted. People do not understand their own brains. (You may agree with this; it is unclear from your wording.) Even if they really did have different desires they would not be mistaken. Even if they used the same sound - ‘right’ - they would be attaching a different meaning to it, so it would be a different word. They would be incorrect if they did not recognize our values as right in Eliezer-speak.
This is admitted a nonintuitive meaning. I do not know if there is a clearer way of saying things and I am unsure of what aspects of most people’s understanding of the word Eliezer believes this to capture. The alternative does not seem much clearer. Consider Eliezer’s example of pulling a child off of some train tracks. If you see me do so, you could explain it in terms of physics/neuroscience. If you ask me about it, I could mention the same explanation, but I also have another one. Why did seeing the child motivate me to save it? Yes, my neural pathways caused it, but I was not thinking about those neural pathway; that would be a level confusion. I was thinking about what is right. Saying that I acted because of neuroscience is true, but saying nothing else promotes level confusion. If you ask me what should happen if I were uninvolved or if my brain were different, I would not change my answer from if I were involved because should is a 1-place function. People do get confused about these things, especially when talking about AI, and that should be stopped. For many people, Eliezer did not resolve confusion, so we need to do better, but default language is no less clear than Eliezer-speak. (To the extent that I agree with Eliezer, I came to this agreement after having read the sequences, but directly after reading other arguments.)
I agree that people don’t fully understand their own brains. I agree that it is possible to have mistaken beliefs about what one really wants. I agree that on EY’s view any group that fails to identify our current values as right is mistaken.
I think EY’s usage of “right” in this context leads to unnecessary confusion.
The alternative that seems clearer to me, as I’ve argued elsewhere, is to designate our values as our values, assert that we endorse our values, engage in research to articulate our values more precisely, build systems to optimize for our values, and evaluate moral arguments in terms of how well they align with our values.
None of this requires further discussion of right and wrong, good and evil, salvatory and diabolical, etc., and such terms seem like “applause lights” better-suited to soliciting alliances than anything else.
If you ask me why I pulled the child off the train tracks, I probably reply that I didn’t want the child to die. If you ask me why I stood on the platform while the train ran over the child, I probably reply that I was paralyzed by shock/fear, or that I wasn’t sure what to do. In both cases, the actual reality is more complicated than my self-report: there are lots of factors that influence what I do, and I’m not aware of most of them.
I agree with you that people get confused about these things. I agree with you that there are multiple levels of description, and mixing them leads to confusion.
If you ask me whether the child should be pulled off the tracks, I probably say “yes”; if you ask me why, I probably get confused. The reason I get confused is because I don’t have a clear understanding of how I come to that conclusion; I simply consulted my preferences.
Faced with that confusion, people make up answers, including answers like “because it’s right to do so” or “because it’s wrong to let the child die” or “because children have moral value” or “because pulling the child off the tracks has shouldness” or a million other such sequences of words, none of which actually help resolve the confusion. They add nothing of value.
There are useful ways to address the question. There are things that can be said about how my preferences came to be that way, and what the consequences are of my preferences being that way, and whether my preferences are consistent. There are techniques for arriving at true statements in those categories.
As far as I can tell, talking about what’s right isn’t among them, any more than talking about what God wants is. It merely adds to the confusion.
I agree with everything non-linguistic If we get rid of words like right, wrong, and should, then we are forced to either come up with new words or use ‘want’ and ‘desire’. The first option is confusing and the second can make us seem like egoists or like people who think that wireheading is right because wireheaded people desire it. To someone unfamiliar with this ethical theory, it would be very misleading. Even many of the readers of this website would be confused if we only used words like ‘want’. What we have now is still far from optimal.
If we get rid of words like right, wrong, and should, then we are forced to either come up with new words or use ‘want’ and ‘desire’.
...and ‘preference’ and ‘value’ and so forth. Yes.
If I am talking about current human values, I endorse calling them that, and avoiding introducing new words (like “right”) until there’s something else for those words to designate.
That neither implies that I’m an egoist, nor that I endorse wireheading.
I agree with you that somebody might nevertheless conclude one or both of those things. They’d be mistaken.
I don’t think familiarity with any particular ethical theory is necessary to interpret the lack of a word, though I agree with you that using a word in the absence of a shared theory about its meaning leads to confusion. (I think most usages of “right” fall into this category.)
If you are using ‘right’ to designate something over and above current human values, I endorse you using the word… but I have no idea at the moment what that something is.
I tentatively agree with your wording, though I will have to see if there are any contexts where it fails.
If you are using ‘right’ to designate something over and above current human values, I endorse you using the word… but I have no idea at the moment what that something is.
By definition, wouldn’t humans be unable to want to pursue such a thing?
For example, if humans value X, and “right” designates Y, and aliens edit our brains so we value Y, then we would want to pursue such a thing. Or if Y is a subset of X, we might find it possible to pursue Y instead of X. (I’m less sure about that, though.) Or various other contrived possibilities.
Yes, my statement was way too strong. In fact, it should be much weaker than even what you say; just start a religion that tells people to value Y. I was attempting to express an actual idea that I had with this sentence originally, but my idea was wrong, so never mind.
But supposing it were true, why would it matter?
What does this mean? Supposing that something were right, what would it matter to humans? You could get it to matter to humans by exploiting their irrationality, but if CEV works, it would not matter to that.
What would it even mean for this to be true? You’d need a definition of right.
Sure.
And if it turns out that humans currently want something different than what we wanted a thousand years ago, then it follows that a thousand years ago we didn’t want what was right, and now we do… though if you’d asked us a thousand years ago, we’d have said that we want what is right, and we’d have arrived at that conclusion through exactly the same cognitive operations we’re currently using. (Of course, in that case we would be mistaken, unlike the current case.)
And if it turns out that a thousand years from now humans want something different, then we will no longer want what is right… though if you ask us then, we’ll say we want what is right, again using the same cognitive operations. (Again, in that case we would be mistaken.)
And if there turn out to be two groups of humans who want incompatible things (for example, because their brains are sufficiently different), then whichever group I happen to be in wants what is right, and the other group doesn’t… though if you ask them, they’ll (mistakenly) say they want what is right, again using the same cognitive operations.
All of which strikes me as a pointlessly confusing way of saying that I endorse what humans-sufficiently-like-me currently want, and don’t endorse what we used to want or come to want or what anyone else wants if it’s too different from that.
Talking about whether some action is right or wrong or moral seems altogether unnecessary on this view. It is enough to say that I endorse what I value, and will program FAI to optimize for that, and will reject moral arguments that are inconsistent with that, and etc. Sure, if I valued something different, I would endorse that instead, but that doesn’t change anything; if I were hit by a speeding train, I’d be dead, but it doesn’t follow that I am dead. I endorse what I value, which means I consider worlds in which there is less of what I value worse than worlds in which there is more of what I value—even if those worlds also include versions of me that endorse something different. Fine and dandy.
What is added to that description by introducing words like right and wrong and moral, other than the confusion caused by people who assume those words refer to a magic light from the sky? It seems no more useful, on this view, than talking about how certain acts are salvatory or diabolical or fleabag.
If the people a thousand years ago might have wanted what is right, but were mistaken as to what they really wanted. People do not understand their own brains. (You may agree with this; it is unclear from your wording.) Even if they really did have different desires they would not be mistaken. Even if they used the same sound - ‘right’ - they would be attaching a different meaning to it, so it would be a different word. They would be incorrect if they did not recognize our values as right in Eliezer-speak.
This is admitted a nonintuitive meaning. I do not know if there is a clearer way of saying things and I am unsure of what aspects of most people’s understanding of the word Eliezer believes this to capture. The alternative does not seem much clearer. Consider Eliezer’s example of pulling a child off of some train tracks. If you see me do so, you could explain it in terms of physics/neuroscience. If you ask me about it, I could mention the same explanation, but I also have another one. Why did seeing the child motivate me to save it? Yes, my neural pathways caused it, but I was not thinking about those neural pathway; that would be a level confusion. I was thinking about what is right. Saying that I acted because of neuroscience is true, but saying nothing else promotes level confusion. If you ask me what should happen if I were uninvolved or if my brain were different, I would not change my answer from if I were involved because should is a 1-place function. People do get confused about these things, especially when talking about AI, and that should be stopped. For many people, Eliezer did not resolve confusion, so we need to do better, but default language is no less clear than Eliezer-speak. (To the extent that I agree with Eliezer, I came to this agreement after having read the sequences, but directly after reading other arguments.)
I agree that people don’t fully understand their own brains. I agree that it is possible to have mistaken beliefs about what one really wants. I agree that on EY’s view any group that fails to identify our current values as right is mistaken.
I think EY’s usage of “right” in this context leads to unnecessary confusion.
The alternative that seems clearer to me, as I’ve argued elsewhere, is to designate our values as our values, assert that we endorse our values, engage in research to articulate our values more precisely, build systems to optimize for our values, and evaluate moral arguments in terms of how well they align with our values.
None of this requires further discussion of right and wrong, good and evil, salvatory and diabolical, etc., and such terms seem like “applause lights” better-suited to soliciting alliances than anything else.
If you ask me why I pulled the child off the train tracks, I probably reply that I didn’t want the child to die. If you ask me why I stood on the platform while the train ran over the child, I probably reply that I was paralyzed by shock/fear, or that I wasn’t sure what to do. In both cases, the actual reality is more complicated than my self-report: there are lots of factors that influence what I do, and I’m not aware of most of them.
I agree with you that people get confused about these things. I agree with you that there are multiple levels of description, and mixing them leads to confusion.
If you ask me whether the child should be pulled off the tracks, I probably say “yes”; if you ask me why, I probably get confused. The reason I get confused is because I don’t have a clear understanding of how I come to that conclusion; I simply consulted my preferences.
Faced with that confusion, people make up answers, including answers like “because it’s right to do so” or “because it’s wrong to let the child die” or “because children have moral value” or “because pulling the child off the tracks has shouldness” or a million other such sequences of words, none of which actually help resolve the confusion. They add nothing of value.
There are useful ways to address the question. There are things that can be said about how my preferences came to be that way, and what the consequences are of my preferences being that way, and whether my preferences are consistent. There are techniques for arriving at true statements in those categories.
As far as I can tell, talking about what’s right isn’t among them, any more than talking about what God wants is. It merely adds to the confusion.
I agree with everything non-linguistic If we get rid of words like right, wrong, and should, then we are forced to either come up with new words or use ‘want’ and ‘desire’. The first option is confusing and the second can make us seem like egoists or like people who think that wireheading is right because wireheaded people desire it. To someone unfamiliar with this ethical theory, it would be very misleading. Even many of the readers of this website would be confused if we only used words like ‘want’. What we have now is still far from optimal.
...and ‘preference’ and ‘value’ and so forth. Yes.
If I am talking about current human values, I endorse calling them that, and avoiding introducing new words (like “right”) until there’s something else for those words to designate.
That neither implies that I’m an egoist, nor that I endorse wireheading.
I agree with you that somebody might nevertheless conclude one or both of those things. They’d be mistaken.
I don’t think familiarity with any particular ethical theory is necessary to interpret the lack of a word, though I agree with you that using a word in the absence of a shared theory about its meaning leads to confusion. (I think most usages of “right” fall into this category.)
If you are using ‘right’ to designate something over and above current human values, I endorse you using the word… but I have no idea at the moment what that something is.
I tentatively agree with your wording, though I will have to see if there are any contexts where it fails.
By definition, wouldn’t humans be unable to want to pursue such a thing?
Not necessarily.
For example, if humans value X, and “right” designates Y, and aliens edit our brains so we value Y, then we would want to pursue such a thing. Or if Y is a subset of X, we might find it possible to pursue Y instead of X. (I’m less sure about that, though.) Or various other contrived possibilities.
But supposing it were true, why would it matter?
Yes, my statement was way too strong. In fact, it should be much weaker than even what you say; just start a religion that tells people to value Y. I was attempting to express an actual idea that I had with this sentence originally, but my idea was wrong, so never mind.
What does this mean? Supposing that something were right, what would it matter to humans? You could get it to matter to humans by exploiting their irrationality, but if CEV works, it would not matter to that.
What would it even mean for this to be true? You’d need a definition of right.