Eliezer: “When a paperclip maximizer and a pencil maximizer do different things, they are not disagreeing about anything, they are just different optimization processes. You cannot detach should-ness from any specific criterion of should-ness and be left with a pure empty should-ness that the paperclip maximizer and pencil maximizer can be said to disagree about—unless you cover “disagreement” to include differences where two agents have nothing to say to each other.
But this would be an extreme position to take with respect to your fellow humans, and I recommend against doing so. Even a psychopath would still be in a common moral reference frame with you, if, fully informed, they would decide to take a pill that would make them non-psychopaths. If you told me that my ability to care about other people was neurologically damaged, and you offered me a pill to fix it, I would take it.”
no, you wouldn’t. The only reason that you are now saying that you would take it is that you currently have the ability to care about other people. Surely this is obvious? Eliezer, you are ignoring your own advice and summoning a “ghost of perfect human unity” into every human mind, even that of a psychopath. Your ability to want to make yourself to care about other people only comes because you already care about other people.
I find your position somewhat outlandish: that every single moral disagreement between humans is simply there because the humans involved in it aren’t fully informed, but that there is no “information” that I (or perhaps someone vastly more intelligent than me) could give to a paperclip maximizer that would persuade them that their ability to love human babies is damaged and they need to take a pill to fix it. (Hat tip to TGGP here) And this coming from the inventor of the AI-box thought experiment!
Let me spell it out. Every human mind comes with an evolved set of “yuck” factors (and their opposite, which I might call “yum” factors?). This is the “psychological unity of humankind”. Unfortunately, these cover only those situations which we were likely to run into in our EEA. Abortion probably did not exist in our EEA: so people have to compare it to something that did. There are two ways to do this—either you think of it as being just like helping a fellow member of your tribe, and become pro-abortion, or you think of it as being infanticide and become anti abortion. Beyond these “yuck” factors, there is no further unity to the moral views of humankind.
In the modern world, people have to make moral choices using their general intelligence, because there aren’t enough “yuck” and “yum” factors around to give guidance on every question. As such, we shouldn’t expect much more moral agreement from humans than from rational (or approximately rational) AIs.
But in my opinion, there is a certain “psychological unity of approximately rational minds”, and I think Eliezer is being irrational to deny this. Why he is overselling the extent to which humans agree, in flagrant contradiction with the facts, and underselling the extent to which all rational agents share common instrumental values, I do not know.
Eliezer: “When a paperclip maximizer and a pencil maximizer do different things, they are not disagreeing about anything, they are just different optimization processes. You cannot detach should-ness from any specific criterion of should-ness and be left with a pure empty should-ness that the paperclip maximizer and pencil maximizer can be said to disagree about—unless you cover “disagreement” to include differences where two agents have nothing to say to each other.
But this would be an extreme position to take with respect to your fellow humans, and I recommend against doing so. Even a psychopath would still be in a common moral reference frame with you, if, fully informed, they would decide to take a pill that would make them non-psychopaths. If you told me that my ability to care about other people was neurologically damaged, and you offered me a pill to fix it, I would take it.”
no, you wouldn’t. The only reason that you are now saying that you would take it is that you currently have the ability to care about other people. Surely this is obvious? Eliezer, you are ignoring your own advice and summoning a “ghost of perfect human unity” into every human mind, even that of a psychopath. Your ability to want to make yourself to care about other people only comes because you already care about other people.
I find your position somewhat outlandish: that every single moral disagreement between humans is simply there because the humans involved in it aren’t fully informed, but that there is no “information” that I (or perhaps someone vastly more intelligent than me) could give to a paperclip maximizer that would persuade them that their ability to love human babies is damaged and they need to take a pill to fix it. (Hat tip to TGGP here) And this coming from the inventor of the AI-box thought experiment!
Let me spell it out. Every human mind comes with an evolved set of “yuck” factors (and their opposite, which I might call “yum” factors?). This is the “psychological unity of humankind”. Unfortunately, these cover only those situations which we were likely to run into in our EEA. Abortion probably did not exist in our EEA: so people have to compare it to something that did. There are two ways to do this—either you think of it as being just like helping a fellow member of your tribe, and become pro-abortion, or you think of it as being infanticide and become anti abortion. Beyond these “yuck” factors, there is no further unity to the moral views of humankind.
In the modern world, people have to make moral choices using their general intelligence, because there aren’t enough “yuck” and “yum” factors around to give guidance on every question. As such, we shouldn’t expect much more moral agreement from humans than from rational (or approximately rational) AIs.
But in my opinion, there is a certain “psychological unity of approximately rational minds”, and I think Eliezer is being irrational to deny this. Why he is overselling the extent to which humans agree, in flagrant contradiction with the facts, and underselling the extent to which all rational agents share common instrumental values, I do not know.