For example, let’s say a self-aware and sentient paperclip maximizer were to be built. Can killing it be said to be unethical?
The maximiser would get a “vote” in the utility calculation, just as every other mind would get a vote. i.e., its preferences are fully considerable. We’re performing an expected utility calculation that weighs each mind’s preferences equally.
So the maximiser has a strong preference not to die, which is a negative on killing it. But assuming it’s going to tile the universe with paperclips, its vote would get out-weighed by the votes of all the other minds.
So the maximiser has a strong preference not to die, which is a negative on killing it. But assuming it’s going to tile the universe with paperclips, its vote would get out-weighed by the votes of all the other minds.
A less convenient possible world: There are a trillion paper-clip maximizers. They prefer the matter we are made out of to be used for paper-clips.
Even though there’s no moral realism, it still seems wrong that such an important ethical question turns out to hinge on whether humans or paper-clip-maximizers started breeding first. One way of not biting that bullet is to say that we shouldn’t be “voting” at all. The only good reason to vote is when there are scarce, poorly divisible resources. For example, it makes sense to vote on what audio tracks to put on the Pioneer satellite; we can only afford to launch, e.g. 100 short sound clips, and making the clips even shorter to accommodate everyone’s preferred tracks would just ruin them for everyone. On the other hand, if five people want to play jump rope and two people want to play hopscotch, the solution isn’t to hold a vote and make everyone play jump rope—the solution is for five people to play jump rope and two people to play hopscotch. Similarly, if 999 billion Clippys want to make paperclips and a billion humans want to build underground volcano lairs, and they both need the same matter to do it, and Clippies experience roughly the same amount of pleasure and pain as humans, then let the Clippies use 99.9% of the galaxy’s matter to build paper clips, and let the humans use 0.1% of the galaxy’s matter to build underground volcano lairs. There’s no need to hold a vote or even to attempt to compare the absolute value of human utility with the absolute value of Clippy utility.
The interesting question is what to do about so-called “utility monsters”—people who, for whatever reason, experience pleasure and pain much more deeply than average. Should their preferences count more? What if they self-modified into utility monsters specifically in order to have their preferences count more? What if they did so in an overtly strategic way, e.g., +20 utility if all demands are met, and −1,000,000 utility if any demands are even slightly unmet? More mundanely, if I credibly pre-commit to being tortured unless I get to pick what kind of pizza we all order, should you give in?
I think Eliezer addressed that at one point (using a cake-making intelligence, I believe) - it would be more ethical, from a human perspective, to allow the paperclippers to make paperclips. However, it would be unethical to change the world from it’s current state to one containing trillions of paperclippers, since the CEV of current people don’t want that.
The maximiser would get a “vote” in the utility calculation, just as every other mind would get a vote. i.e., its preferences are fully considerable. We’re performing an expected utility calculation that weighs each mind’s preferences equally.
(Trying to understand your intuitions better. I’m considering positions similar to yours and so maybe I can let you do some of the thinking for me.)
Is there a way for a mind A to prefer something more in an ethical sense than mind B? Is there some kind of Occam’s Razor of preferences, say based on complexity (wanting everything to be turned into paperclips is better than wanting everything to be turned into Japanese cast productions of Shakespeare in the original Klingon) or some kind of resource (wanting one more paperclip is better than wanting a universe full)?
How do you handle different preferences in one agent? Say a human wants to both eat chocolate ice cream and ban it. Do you treat this as two agents? (Based on what? Temporal separation? Spatial? Logical contradictions?)
Let’s say a paperclipper comes along and wants to turn us all into paperclips. Let’s limit it to the paperclipper vs. us humans for simplicity’s sake, and let’s assume we are all perfectly ethical. So we do what? Have a vote? Which kind of voting system (and why)? What if we don’t trust some party—who does the counting? (In other words, how do you determine the preferences of an uncooperative agent?)
(Doesn’t unconditionally including other minds into your moral calculus open you up to all kinds of exploits? Say I want to do X and you want NOT-X. So I breed and make a hundred more agents that want X. No matter how strong and unwilling you are, I can force you to cooperate with me, as long as there’s enough minds on my side and you are bound to act ethically.)
Similarly, how do you deal with a hive-mind? Let’s say Anonymous states tomorrow that it wants otters to be removed from the universe. Is this one vote? As many as Anonymous has members? (… that have been participating in the formulation or in general?) (Analogously for brains.)
The maximiser would get a “vote” in the utility calculation, just as every other mind would get a vote. i.e., its preferences are fully considerable. We’re performing an expected utility calculation that weighs each mind’s preferences equally.
So the maximiser has a strong preference not to die, which is a negative on killing it. But assuming it’s going to tile the universe with paperclips, its vote would get out-weighed by the votes of all the other minds.
A less convenient possible world: There are a trillion paper-clip maximizers. They prefer the matter we are made out of to be used for paper-clips.
Then it’s more ethical to give the maximisers what they want.
Even though there’s no moral realism, it still seems wrong that such an important ethical question turns out to hinge on whether humans or paper-clip-maximizers started breeding first. One way of not biting that bullet is to say that we shouldn’t be “voting” at all. The only good reason to vote is when there are scarce, poorly divisible resources. For example, it makes sense to vote on what audio tracks to put on the Pioneer satellite; we can only afford to launch, e.g. 100 short sound clips, and making the clips even shorter to accommodate everyone’s preferred tracks would just ruin them for everyone. On the other hand, if five people want to play jump rope and two people want to play hopscotch, the solution isn’t to hold a vote and make everyone play jump rope—the solution is for five people to play jump rope and two people to play hopscotch. Similarly, if 999 billion Clippys want to make paperclips and a billion humans want to build underground volcano lairs, and they both need the same matter to do it, and Clippies experience roughly the same amount of pleasure and pain as humans, then let the Clippies use 99.9% of the galaxy’s matter to build paper clips, and let the humans use 0.1% of the galaxy’s matter to build underground volcano lairs. There’s no need to hold a vote or even to attempt to compare the absolute value of human utility with the absolute value of Clippy utility.
The interesting question is what to do about so-called “utility monsters”—people who, for whatever reason, experience pleasure and pain much more deeply than average. Should their preferences count more? What if they self-modified into utility monsters specifically in order to have their preferences count more? What if they did so in an overtly strategic way, e.g., +20 utility if all demands are met, and −1,000,000 utility if any demands are even slightly unmet? More mundanely, if I credibly pre-commit to being tortured unless I get to pick what kind of pizza we all order, should you give in?
I think Eliezer addressed that at one point (using a cake-making intelligence, I believe) - it would be more ethical, from a human perspective, to allow the paperclippers to make paperclips. However, it would be unethical to change the world from it’s current state to one containing trillions of paperclippers, since the CEV of current people don’t want that.
(Trying to understand your intuitions better. I’m considering positions similar to yours and so maybe I can let you do some of the thinking for me.)
Is there a way for a mind A to prefer something more in an ethical sense than mind B? Is there some kind of Occam’s Razor of preferences, say based on complexity (wanting everything to be turned into paperclips is better than wanting everything to be turned into Japanese cast productions of Shakespeare in the original Klingon) or some kind of resource (wanting one more paperclip is better than wanting a universe full)?
How do you handle different preferences in one agent? Say a human wants to both eat chocolate ice cream and ban it. Do you treat this as two agents? (Based on what? Temporal separation? Spatial? Logical contradictions?)
Let’s say a paperclipper comes along and wants to turn us all into paperclips. Let’s limit it to the paperclipper vs. us humans for simplicity’s sake, and let’s assume we are all perfectly ethical. So we do what? Have a vote? Which kind of voting system (and why)? What if we don’t trust some party—who does the counting? (In other words, how do you determine the preferences of an uncooperative agent?)
(Doesn’t unconditionally including other minds into your moral calculus open you up to all kinds of exploits? Say I want to do X and you want NOT-X. So I breed and make a hundred more agents that want X. No matter how strong and unwilling you are, I can force you to cooperate with me, as long as there’s enough minds on my side and you are bound to act ethically.)
Similarly, how do you deal with a hive-mind? Let’s say Anonymous states tomorrow that it wants otters to be removed from the universe. Is this one vote? As many as Anonymous has members? (… that have been participating in the formulation or in general?) (Analogously for brains.)