Let’s pretend that you are a utilitarian. You want to satisfy everyone’s goals
This isn’t a criticism of the substance of your argument, but I’ve come across a view like this one frequently on LW so I want to address it: This seems like a pretty nonstandard definition of “utilitarian,” or at least, it’s only true of some kinds of preference utilitarianism.
I think utilitarianism usually refers to a view where what you ought to do is maximize a utility function that (somehow) aggregates a metric of welfare across individuals, not their goal-satisfaction. Kicking a puppy without me knowing about it thwarts my goals, but (at least on many reasonable conceptions of “welfare”) doesn’t decrease my welfare.
I’d be very surprised if most utilitarians thought they’d have a moral obligation to create paperclips if 99.99% of agents in the world were paperclippers (example stolen from Brian Tomasik), controlling for game-theoretic instrumental reasons.
I think in the social choice literature, people almost always mean preference utilitarianism when they say “utilitarianism”, whereas in the philosophical/ethics literature people are more likely to mean hedonic utilitarianism. I think the reason for this is that in the social choice and somewhat adjacent game (and decision) theory literature, utility functions have a fairly solid foundation as a representation of preferences of rational agents. (For example, Harsanyi’s “[preference] utilitarian theorem” paper and Nash’s paper on the Nash bargaining solution make very explicit reference to this foundation.) Whereas there is no solid foundation for numeric hedonic welfare (at least not in this literature, but also not elsewhere as far as I know).
Agreed. I should have had disclaimer that I was talking about preference utilitarianism.
I am not sure what is true about what most people think.
My guess is that most philosophers who identify with utilitarianism mean welfare.
I would guess that most readers of LessWrong would not identify with utilitarianism, but would say they identify more with preference utilitarianism than welfare utilitarianism.
My guess is that a larger (relative to LW) proportion of EAs identify with utilitarianism, and also they identify with the welfare version (relative to preference version) more than LW, but I have a lot of uncertainty about how much. (There is probably some survey data that could answer this question. I haven’t checked.)
Also, I am not sure that “controlling for game-theoretic instrumental reasons” is actually a move that is well defined/makes sense.
I am not sure that “controlling for game-theoretic instrumental reasons” is actually a move that is well defined/makes sense.
I don’t have a crisp definition of this, but I just mean that, e.g., we compare the following two worlds: (1) 99.99% of agents are non-sentient paperclippers, and each agent has equal (bargaining) power. (2) 99.99% of agents are non-sentient paperclippers, and the paperclippers are all confined to some box. According to plenty of intuitive-to-me value systems, you only (maybe) have reason to increase paperclips in (1), not (2). But if the paperclippers felt really sad about the world not having more paperclips, I’d care—to an extent that depends on the details of the situation—about increasing paperclips even in (2).
This isn’t a criticism of the substance of your argument, but I’ve come across a view like this one frequently on LW so I want to address it: This seems like a pretty nonstandard definition of “utilitarian,” or at least, it’s only true of some kinds of preference utilitarianism.
I think utilitarianism usually refers to a view where what you ought to do is maximize a utility function that (somehow) aggregates a metric of welfare across individuals, not their goal-satisfaction. Kicking a puppy without me knowing about it thwarts my goals, but (at least on many reasonable conceptions of “welfare”) doesn’t decrease my welfare.
I’d be very surprised if most utilitarians thought they’d have a moral obligation to create paperclips if 99.99% of agents in the world were paperclippers (example stolen from Brian Tomasik), controlling for game-theoretic instrumental reasons.
I think in the social choice literature, people almost always mean preference utilitarianism when they say “utilitarianism”, whereas in the philosophical/ethics literature people are more likely to mean hedonic utilitarianism. I think the reason for this is that in the social choice and somewhat adjacent game (and decision) theory literature, utility functions have a fairly solid foundation as a representation of preferences of rational agents. (For example, Harsanyi’s “[preference] utilitarian theorem” paper and Nash’s paper on the Nash bargaining solution make very explicit reference to this foundation.) Whereas there is no solid foundation for numeric hedonic welfare (at least not in this literature, but also not elsewhere as far as I know).
Agreed. I should have had disclaimer that I was talking about preference utilitarianism.
I am not sure what is true about what most people think.
My guess is that most philosophers who identify with utilitarianism mean welfare.
I would guess that most readers of LessWrong would not identify with utilitarianism, but would say they identify more with preference utilitarianism than welfare utilitarianism.
My guess is that a larger (relative to LW) proportion of EAs identify with utilitarianism, and also they identify with the welfare version (relative to preference version) more than LW, but I have a lot of uncertainty about how much. (There is probably some survey data that could answer this question. I haven’t checked.)
Also, I am not sure that “controlling for game-theoretic instrumental reasons” is actually a move that is well defined/makes sense.
I agree with your guesses.
I don’t have a crisp definition of this, but I just mean that, e.g., we compare the following two worlds: (1) 99.99% of agents are non-sentient paperclippers, and each agent has equal (bargaining) power. (2) 99.99% of agents are non-sentient paperclippers, and the paperclippers are all confined to some box. According to plenty of intuitive-to-me value systems, you only (maybe) have reason to increase paperclips in (1), not (2). But if the paperclippers felt really sad about the world not having more paperclips, I’d care—to an extent that depends on the details of the situation—about increasing paperclips even in (2).