The maximiser would get a “vote” in the utility calculation, just as every other mind would get a vote. i.e., its preferences are fully considerable. We’re performing an expected utility calculation that weighs each mind’s preferences equally.
(Trying to understand your intuitions better. I’m considering positions similar to yours and so maybe I can let you do some of the thinking for me.)
Is there a way for a mind A to prefer something more in an ethical sense than mind B? Is there some kind of Occam’s Razor of preferences, say based on complexity (wanting everything to be turned into paperclips is better than wanting everything to be turned into Japanese cast productions of Shakespeare in the original Klingon) or some kind of resource (wanting one more paperclip is better than wanting a universe full)?
How do you handle different preferences in one agent? Say a human wants to both eat chocolate ice cream and ban it. Do you treat this as two agents? (Based on what? Temporal separation? Spatial? Logical contradictions?)
Let’s say a paperclipper comes along and wants to turn us all into paperclips. Let’s limit it to the paperclipper vs. us humans for simplicity’s sake, and let’s assume we are all perfectly ethical. So we do what? Have a vote? Which kind of voting system (and why)? What if we don’t trust some party—who does the counting? (In other words, how do you determine the preferences of an uncooperative agent?)
(Doesn’t unconditionally including other minds into your moral calculus open you up to all kinds of exploits? Say I want to do X and you want NOT-X. So I breed and make a hundred more agents that want X. No matter how strong and unwilling you are, I can force you to cooperate with me, as long as there’s enough minds on my side and you are bound to act ethically.)
Similarly, how do you deal with a hive-mind? Let’s say Anonymous states tomorrow that it wants otters to be removed from the universe. Is this one vote? As many as Anonymous has members? (… that have been participating in the formulation or in general?) (Analogously for brains.)
(Trying to understand your intuitions better. I’m considering positions similar to yours and so maybe I can let you do some of the thinking for me.)
Is there a way for a mind A to prefer something more in an ethical sense than mind B? Is there some kind of Occam’s Razor of preferences, say based on complexity (wanting everything to be turned into paperclips is better than wanting everything to be turned into Japanese cast productions of Shakespeare in the original Klingon) or some kind of resource (wanting one more paperclip is better than wanting a universe full)?
How do you handle different preferences in one agent? Say a human wants to both eat chocolate ice cream and ban it. Do you treat this as two agents? (Based on what? Temporal separation? Spatial? Logical contradictions?)
Let’s say a paperclipper comes along and wants to turn us all into paperclips. Let’s limit it to the paperclipper vs. us humans for simplicity’s sake, and let’s assume we are all perfectly ethical. So we do what? Have a vote? Which kind of voting system (and why)? What if we don’t trust some party—who does the counting? (In other words, how do you determine the preferences of an uncooperative agent?)
(Doesn’t unconditionally including other minds into your moral calculus open you up to all kinds of exploits? Say I want to do X and you want NOT-X. So I breed and make a hundred more agents that want X. No matter how strong and unwilling you are, I can force you to cooperate with me, as long as there’s enough minds on my side and you are bound to act ethically.)
Similarly, how do you deal with a hive-mind? Let’s say Anonymous states tomorrow that it wants otters to be removed from the universe. Is this one vote? As many as Anonymous has members? (… that have been participating in the formulation or in general?) (Analogously for brains.)