Even though there’s no moral realism, it still seems wrong that such an important ethical question turns out to hinge on whether humans or paper-clip-maximizers started breeding first. One way of not biting that bullet is to say that we shouldn’t be “voting” at all. The only good reason to vote is when there are scarce, poorly divisible resources. For example, it makes sense to vote on what audio tracks to put on the Pioneer satellite; we can only afford to launch, e.g. 100 short sound clips, and making the clips even shorter to accommodate everyone’s preferred tracks would just ruin them for everyone. On the other hand, if five people want to play jump rope and two people want to play hopscotch, the solution isn’t to hold a vote and make everyone play jump rope—the solution is for five people to play jump rope and two people to play hopscotch. Similarly, if 999 billion Clippys want to make paperclips and a billion humans want to build underground volcano lairs, and they both need the same matter to do it, and Clippies experience roughly the same amount of pleasure and pain as humans, then let the Clippies use 99.9% of the galaxy’s matter to build paper clips, and let the humans use 0.1% of the galaxy’s matter to build underground volcano lairs. There’s no need to hold a vote or even to attempt to compare the absolute value of human utility with the absolute value of Clippy utility.
The interesting question is what to do about so-called “utility monsters”—people who, for whatever reason, experience pleasure and pain much more deeply than average. Should their preferences count more? What if they self-modified into utility monsters specifically in order to have their preferences count more? What if they did so in an overtly strategic way, e.g., +20 utility if all demands are met, and −1,000,000 utility if any demands are even slightly unmet? More mundanely, if I credibly pre-commit to being tortured unless I get to pick what kind of pizza we all order, should you give in?
Even though there’s no moral realism, it still seems wrong that such an important ethical question turns out to hinge on whether humans or paper-clip-maximizers started breeding first. One way of not biting that bullet is to say that we shouldn’t be “voting” at all. The only good reason to vote is when there are scarce, poorly divisible resources. For example, it makes sense to vote on what audio tracks to put on the Pioneer satellite; we can only afford to launch, e.g. 100 short sound clips, and making the clips even shorter to accommodate everyone’s preferred tracks would just ruin them for everyone. On the other hand, if five people want to play jump rope and two people want to play hopscotch, the solution isn’t to hold a vote and make everyone play jump rope—the solution is for five people to play jump rope and two people to play hopscotch. Similarly, if 999 billion Clippys want to make paperclips and a billion humans want to build underground volcano lairs, and they both need the same matter to do it, and Clippies experience roughly the same amount of pleasure and pain as humans, then let the Clippies use 99.9% of the galaxy’s matter to build paper clips, and let the humans use 0.1% of the galaxy’s matter to build underground volcano lairs. There’s no need to hold a vote or even to attempt to compare the absolute value of human utility with the absolute value of Clippy utility.
The interesting question is what to do about so-called “utility monsters”—people who, for whatever reason, experience pleasure and pain much more deeply than average. Should their preferences count more? What if they self-modified into utility monsters specifically in order to have their preferences count more? What if they did so in an overtly strategic way, e.g., +20 utility if all demands are met, and −1,000,000 utility if any demands are even slightly unmet? More mundanely, if I credibly pre-commit to being tortured unless I get to pick what kind of pizza we all order, should you give in?