Population Ethics Shouldn’t Be About Maximizing Utility
let me suggest a moral axiom with apparently very strong intuitive support, no matter what your concept of morality: morality should exist. That is, there should exist creatures who know what is moral, and who act on that. So if your moral theory implies that in ordinary circumstances moral creatures should exterminate themselves, leaving only immoral creatures, or no creatures at all, well that seems a sufficient reductio to solidly reject your moral theory.
I agree strongly with the above quote, and I think most other readers will as well. It is good for moral beings to exist and a world with beings who value morality is almost always better than one where they do not. I would like to restate this more precisely as the following axiom: A population in which moral beings exist and have net positive utility, and in which all other creatures in existence also have net positive utility, is always better than a population where moral beings do not exist.
While the axiom that morality should exist is extremely obvious to most people, there is one strangely popular ethical system that rejects it: total utilitarianism. In this essay I will argue that Total Utilitarianism leads to what I will call the Genocidal Conclusion, which is that there are many situations in which it would be fantastically good for moral creatures to either exterminate themselves, or greatly limit their utility and reproduction in favor of the utility and reproduction of immoral creatures. I will argue that the main reason consequentialist theories of population ethics produce such obviously absurd conclusions is that they continue to focus on maximizing utility1 in situations where it is possible to create new creatures. I will argue that pure utility maximization is only a valid ethical theory for “special case” scenarios where the population is static. I will propose an alternative theory for population ethics I call “ideal consequentialism” or “ideal utilitarianism” which avoids the Genocidal Conclusion and may also avoid the more famous Repugnant Conclusion.
I will begin my argument by pointing to a common problem in population ethics known as the Mere Addition Paradox (MAP) and the Repugnant Conclusion. Most Less Wrong readers will already be familiar with this problem, so I do not think I need to elaborate on it. You may also be familiar with a even stronger variation called the Benign Addition Paradox (BAP). This is essentially the same as the MAP, except that each time one adds more people one also gives a small amount of additional utility to the people who already existed. One then proceeds to redistribute utility between people as normal, eventually arriving at the huge population where everyone’s lives are “barely worth living.” The point of this is to argue that the Repugnant Conclusion can be arrived at from “mere addition” of new people that not only doesn’t harm the preexisting-people, but also one that benefits them.
The next step of my argument involves three slightly tweaked versions of the Benign Addition Paradox. I have not changed the basic logic of the problem, I have just added one small clarifying detail. In the original MAP and BAP it was not specified what sort of values the added individuals in population A+ held. Presumably one was meant to assume that they were ordinary human beings. In the versions of the BAP I am about to present, however, I will specify that the extra individuals added in A+ are not moral creatures, that if they have values at all they are values indifferent to, or opposed to, morality and the other values that the human race holds dear.
1. The Benign Addition Paradox with Paperclip Maximizers.
Let us imagine, as usual, a population, A, which has a large group of human beings living lives of very high utility. Let us then add a new population consisting of paperclip maximizers, each of whom is living a life barely worth living. Presumably, for a paperclip maximizer, this would be a life where the paperclip maximizer’s existence results in at least one more paperclip in the world than there would have been otherwise.
Now, one might object that if one creates a paperclip maximizer, and then allows it to create one paperclip, the utility of the other paperclip maximizers will increase above the “barely worth living” level, which would obviously make this thought experiment nonalagous with the original MAP and BAP. To prevent this we will assume that each paperclip maximizer that is created has a slightly different values on what the ideal size, color, and composition of the paperclip they are trying to produce is. So the Purple 2 centimeter Plastic Paperclip Maximizer gains no addition utility from when the Silver Iron 1 centimeter Paperclip Maximizer makes a paperclip.
So again, let us add these paperclip maximizers to population A, and in the process give one extra utilon of utility to each preexisting person in A. This is a good thing, right? After all, everyone in A benefited, and the paperclippers get to exist and make paperclips. So clearly A+, the new population, is better than A.
Now let’s take the next step, the transition from population A+ to population B. Take some of the utility from the human beings and convert it into paperclips. This is a good thing, right?
So let us repeat these steps adding paperclip maximizers and utility, and then redistributing utility. Eventually we reach population Z, where there is a vast amount of paperclip maximizers, a vast amount of many different kinds of paperclips, and a small amount of human beings living lives barely worth living.
Obviously Z is better than A, right? We should not fear the creation of a paperclip maximizing AI, but welcome it! Forget about things like high challenge, love, interpersonal entanglement, complex fun, and so on! Those things just don’t produce the kind of utility that paperclip maximization has the potential to do!
Or maybe there is something seriously wrong with the moral assumptions behind the Mere Addition and Benign Addition Paradoxes.
But you might argue that I am using an unrealistic example. Creatures like Paperclip Maximizers may be so far removed from normal human experience that we have trouble thinking about them properly. So let’s replay the Benign Addition Paradox again, but with creatures we might actually expect to meet in real life, and we know we actually value.
2. The Benign Addition Paradox with Non-Sapient Animals
You know the drill by now. Take population A, add a new population to it, while very slightly increasing the utility of the original population. This time let’s have it be some kind animal that is capable of feeling pleasure and pain, but is not capable of modeling possible alternative futures and choosing between them (in other words, it is not capable of having “values” or being “moral”). A lizard or a mouse, for example. Each one feels slightly more pleasure than pain in its lifetime, so it can be said to have a life barely worth living. Convert A+ to B. Take the utilons that the human beings are using to experience things like curiosity, beatitude, wisdom, beauty, harmony, morality, and so on, and convert it into pleasure for the animals.
We end up with population Z, with a vast amount of mice or lizards with lives just barely worth living, and a small amount of human beings with lives barely worth living. Terrific! Why do we bother creating humans at all! Let’s just create tons of mice and inject them full of heroin! It’s a much more efficient way to generate utility!
3. The Benign Addition Paradox with Sociopaths
What new population will we add to A this time? How about some other human beings, who all have anti-social personality disorder? True, they lack the key, crucial value of sympathy that defines so much of human behavior. But they don’t seem to miss it. And their lives are barely worth living, so obviously A+ has greater utility than A. If given a chance the sociopaths will reduce the utility of other people to negative levels, but let’s assume that that is somehow prevented in this case.
Eventually we get to Z, with a vast population of sociopaths and a small population of normal human beings, all living lives just barely worth living. That has more utility, right? True, the sociopaths place no value on things like friendship, love, compassion, empathy, and so on. And true, the sociopaths are immoral beings who do not care in the slightest about right and wrong. But what does that matter? Utility is being maximized, and surely that is what population ethics is all about!
Asteroid!
Let’s suppose an asteroid is approaching each of the four population Zs discussed before. It can only be deflected by so much. Your choice is, save the original population of humans from A, or save the vast new population. The choice is obvious. In 1, 2, and 3, each individual has the same level utility, so obviously we should choose which option saves a greater number of individuals.
Bam! The asteroid strikes. The end result in all four scenarios is a world in which all the moral creatures are destroyed. It is a world without the many complex values that human beings possess. Each world, for the most part, lack things like complex challenge, imagination, friendship, empathy, love, and the other complex values that human beings prize. But so what? The purpose of population ethics is to maximize utility, not silly, frivolous things like morality, or the other complex values of the human race. That means that any form of utility that is easier to produce than those values is obviously superior. It’s easier to make pleasure and paperclips than it is to make eudaemonia, so that’s the form of utility that ought to be maximized, right? And as for making sure moral beings exist, well that’s just ridiculous. The valuable processing power they’re using to care about morality could be being used to make more paperclips or more mice injected with heroin! Obviously it would be better if they died off, right?
I’m going to go out on a limb and say “Wrong.”
Is this realistic?
Now, to fair, in the Overcoming Bias page I quoted, Robin Hanson also says:
I’m not saying I can’t imagine any possible circumstances where moral creatures shouldn’t die off, but I am saying that those are not ordinary circumstances.
Maybe the scenarios I am proposing are just too extraordinary. But I don’t think this is the case. I imagine that the circumstances Robin had in mind were probably something like “either all moral creatures die off, or all moral creatures are tortured 24⁄7 for all eternity.”
Any purely utility-maximizing theory of population ethics that counts both the complex values of human beings, and the pleasure of animals, as “utility” should inevitably draw the conclusion that human beings ought to limit their reproduction to the bare minimum necessary to maintain the infrastructure to sustain a vastly huge population of non-human animals (preferably animals dosed with some sort of pleasure-causing drug). And if some way is found to maintain that infrastructure automatically, without the need for human beings, then the logical conclusion is that human beings are a waste of resources (as are chimps, gorillas, dolphins, and any other animal that is even remotely capable of having values or morality). Furthermore, even if the human race cannot practically be replaced with automated infrastructure, this should be an end result that the adherents of this theory should be yearning for.2 There should be much wailing and gnashing of teeth among moral philosophers that exterminating the human race is impractical, and much hope that someday in the future it will not be.
I call this the “Genocidal Conclusion” or “GC.” On the macro level the GC manifests as the idea that the human race ought to be exterminated and replaced with creatures whose preferences are easier to satisfy. On the micro level it manifests as the idea that it is perfectly acceptable to kill someone who is destined to live a perfectly good and worthwhile life and replace them with another person who would have a slightly higher level of utility.
Population Ethics isn’t About Maximizing Utility
I am going to make a rather radical proposal. I am going to argue that the consequentialist’s favorite maxim, “maximize utility,” only applies to scenarios where creating new people or creatures is off the table. I think we need an entirely different ethical framework to describe what ought to be done when it is possible to create new people. I am not by any means saying that “which option would result in more utility” is never a morally relevant consideration when deciding to create a new person, but I definitely think it is not the only one.3
So what do I propose as a replacement to utility maximization? I would argue in favor of a system that promotes a wide range of ideals. Doing some research, I discovered that G. E. Moore had in fact proposed a form of “ideal utilitarianism” in the early 20th century.4 However, I think that “ideal consequentialism” might be a better term for this system, since it isn’t just about aggregating utility functions.
What are some of the ideals that an ideal consequentialist theory of population ethics might seek to promote? I’ve already hinted at what I think they are: Life, consciousness, and activity; health and strength; pleasures and satisfactions of all or certain kinds; happiness, beatitude, contentment, etc.; truth; knowledge and true opinions of various kinds, understanding, wisdom… mutual affection, love, friendship, cooperation; all those other important human universals, plus all the stuff in the Fun Theory Sequence. When considering what sort of creatures to create we ought to create creatures that value those things. Not necessarily, all of them, or in the same proportions, for diversity is an important ideal as well, but they should value a great many of those ideals.
Now, lest you worry that this theory has any totalitarian implications, let me make it clear that I am not saying we should force these values on creatures that do not share them. Forcing a paperclip maximizer to pretend to make friends and love people does not do anything to promote the ideals of Friendship and Love. Forcing a chimpanzee to listen while you read the Sequences to it does not promote the values of Truth and Knowledge. Those ideals require both a subjective and objective component. The only way to promote those ideals is to create a creature that includes them as part of its utility function and then help it maximize its utility.
I am also certainly not saying that there is never any value in creating a creature that does not possess these values. There are obviously many circumstances where it is good to create nonhuman animals. There may even be some circumstances where a paperclip maximizer could be of value. My argument is simply that it is most important to make sure that creatures who value these various ideals exist.
I am also not suggesting that it is morally acceptable to casually inflict horrible harms upon a creature with non-human values if we screw up and create one by accident. If promoting ideals and maximizing utility are separate values then it may be that once we have created such a creature we have a duty to make sure it lives a good life, even if it was a bad thing to create it in the first place. You can’t unbirth a child.5
It also seems to me that in addition to having ideals about what sort of creatures should exist, we also have ideals about how utility ought to be concentrated. If this is the case then ideal consequentialism may be able to block some forms of the Repugnant Conclusion, even if situations where the only creatures whose creation is being considered are human beings. If it is acceptable to create humans instead of paperclippers, even if the paperclippers would have higher utility, it may also be acceptable to create ten humans with a utility of ten each instead of a hundred humans with a utility of 1.01 each.
Why Did We Become Convinced that Maximizing Utility was the Sole Good?
Population ethics was, until comparatively recently, a fallow field in ethics. And in situations where there is no option to increase the population, maximizing utility is the only consideration that’s really relevant. If you’ve created creatures that value the right ideals, then all that is left to be done is to maximize their utility. If you’ve created creatures that do not value the right ideals, there is no value to be had in attempting to force them to embrace those ideals. As I’ve said before, you will not promote the values of Love and Friendship by creating a paperclip maximizer and forcing it to pretend to love people and make friends.
So in situations where the population is constant, “maximize utility” is a decent approximation of the meaning of right. It’s only when the population can be added to that morality becomes much more complicated.
Another thing to blame is human-centric reasoning. When people defend the Repugnant Conclusion they tend to point out that a life barely worth living is not as bad as it would seem at first glance. They emphasize that it need not be a boring life, it may be a life full of ups and downs where the ups just barely outweigh the downs. A life worth living, they say, is a life one would choose to live. Derek Parfit developed this idea to some extent by arguing that there are certain values that are “discontinuous” and that one needs to experience many of them in order to truly have a life worth living.
The Orthogonality Thesis throws all these arguments out the window. It is possible to create an intelligence to execute any utility function, no matter what it is. If human beings have all sorts of complex needs that must be fulfilled in order to for them lead worthwhile lives, then you could create more worthwhile lives by killing the human race and replacing them with something less finicky. Maybe happy cows. Maybe paperclip maximizers. Or how about some creature whose only desire is to live for one second and then die. If we created such a creature and then killed it we would reap huge amounts of utility, for we would have created a creature that got everything it wanted out of life!
How Intuitive is the Mere Addition Principle, Really?
I think most people would agree that morality should exist, and that therefore any system of population ethics should not lead to the Genocidal Conclusion. But which step in the Benign Addition Paradox should we reject? We could reject the step where utility is redistributed. But that seems wrong, most people seem to consider it bad for animals and sociopaths to suffer, and that it is acceptable to inflict at least some amount of disutilities on human beings to prevent such suffering.
It seems more logical to reject the Mere Addition Principle. In other words, maybe we ought to reject the idea that the mere addition of more lives-worth-living cannot make the world worse. And in turn, we should probably also reject the Benign Addition Principle. Adding more lives-worth-living may be capable of making the world worse, even if doing so also slightly benefits existing people. Fortunately this isn’t a very hard principle to reject. While many moral philosophers treat it as obviously correct, nearly everyone else rejects this principle in day-to-day life.
Now, I’m obviously not saying that people’s behavior in their day-to-day lives is always good, it may be that they are morally mistaken. But I think the fact that so many people seem to implicitly reject it provides some sort of evidence against it.
Take people’s decision to have children. Many people choose to have fewer children than they otherwise would because they do not believe they will be able to adequately care for them, at least not without inflicting large disutilities on themselves. If most people accepted the Mere Addition Principle there would be a simple solution for this: have more children and then neglect them! True, the children’s lives would be terrible while they were growing up, but once they’ve grown up and are on their own there’s a good chance they may be able to lead worthwhile lives. Not only that, it may be possible to trick the welfare system into giving you money for the children you neglect, which would satisfy the Benign Addition Principle.
Yet most people choose not to have children and neglect them. And furthermore they seem to think that they have a moral duty not to do so, that a world where they choose to not have neglected children is better than one that they don’t. What is wrong with them?
Another example is a common political view many people have. Many people believe that impoverished people should have fewer children because of the burden doing so would place on the welfare system. They also believe that it would be bad to get rid of the welfare system altogether. If the Benign Addition Principle were as obvious as it seems, they would instead advocate for the abolition of the welfare system, and encourage impoverished people to have more children. Assuming most impoverished people live lives worth living, this is exactly analogous to the BAP, it would create more people, while benefiting existing ones (the people who pay less taxes because of the abolition of the welfare system).
Yet again, most people choose to reject this line of reasoning. The BAP does not seem to be an obvious and intuitive principle at all.
The Genocidal Conclusion is Really Repugnant
There is nearly nothing repugnant than the Genocidal Conclusion. Pretty much the only way a line of moral reasoning could go more wrong would be concluding that we have a moral duty to cause suffering, as an end in itself. This means that it’s fairly easy to counter any argument in favor of total utilitarianism that argues the alternative I am promoting has odd conclusions that do not fit some of our moral intuitions, while total utilitarianism does not. Is that conclusion more insane than the Genocidal Conclusion? If it isn’t, total utilitarianism should still be rejected.
Ideal Consequentialism Needs a Lot of Work
I do think that Ideal Consequentialism needs some serious ironing out. I haven’t really developed it into a logical and rigorous system, at this point it’s barely even a rough framework. There are many questions that stump me. In particular I am not quite sure what population principle I should develop. It’s hard to develop one that rejects the MAP without leading to weird conclusions, like that it’s bad to create someone of high utility if a population of even higher utility existed long ago. It’s a difficult problem to work on, and it would be interesting to see if anyone else had any ideas.
But just because I don’t have an alternative fully worked out doesn’t mean I can’t reject Total Utilitarianism. It leads to the conclusion that a world with no love, curiosity, complex challenge, friendship, morality, or any other value the human race holds dear is an ideal, desirable world, if there is a sufficient amount of some other creature with a simpler utility function. Morality should exist, and because of that, total utilitarianism must be rejected as a moral system.
1I have been asked to note that when I use the phrase “utility” I am usually referring to a concept that is called “E-utility,” rather than the Von Neumann-Morgenstern utility that is sometimes discussed in decision theory. The difference is that in VNM one’s moral views are included in one’s utility function, whereas in E-utility they are not. So if one chooses to harm oneself to help others because one believes that is morally right, one has higher VNM utility, but lower E-utility.
2There is a certain argument against the Repugnant Conclusion that goes that, as the steps of the Mere Addition Paradox are followed the world will lose its last symphony, its last great book, and so on. I have always considered this to be an invalid argument because the world of the RC doesn’t necessarily have to be one where these things don’t exist, it could be one where they exist, but are enjoyed very rarely. The Genocidal Conclusion brings this argument back in force. Creating creatures that can appreciate symphonies and great books is very inefficient compared to creating bunny rabbits pumped full of heroin.
3Total Utilitarianism was originally introduced to population ethics as a possible solution to the Non-Identity Problem. I certainly agree that such a problem needs a solution, even if Total Utilitarianism doesn’t work out as that solution.
4I haven’t read a lot of Moore, most of my ideas were extrapolated from other things I read on Less Wrong. I just mentioned him because in my research I noticed his concept of “ideal utilitarianism” resembled my ideas. While I do think he was on the right track he does commit the Mind Projection Fallacy a lot. For instance, he seems to think that one could promote beauty by creating beautiful objects, even if there were no creatures with standards of beauty around to appreciate them. This is why I am careful to emphasize that to promote ideals like love and beauty one must create creatures capable of feeling love and experiencing beauty.
5My tentative answer to the question Eliezer poses in “You Can’t Unbirth a Child” is that human beings may have a duty to allow the cheesecake maximizers to build some amount of giant cheesecakes, but they would also have a moral duty to limit such creatures’ reproduction in order to spare resources to create more creatures with humane values.
EDITED: To make a point about ideal consequentialism clearer, based on AlexMennen’s criticisms.
What does it even mean for a creature not to have “morals” and yet to have goals (often competing ones) that it tries to pursue?
Why wouldn’t it be possible to arbitrarily relabel any of those goals as ‘morals’? How would you restrict morals such that they are not conceivably identical with any part of a utility function, without being overly anthropocentric in what you allow as a valid moral code?
You can, but I think you’ve lost a distinction that most of us make. We have preferences. Some of those preferences we call moral. Some we don’t. Losing that distinction loses information.
You may prefer vanilla over chocolate, but few would call that a moral preference. One part of the distinction is that ice cream flavor preference has no 3rd person or higher order preferences. We don’t disapprove of people for eating the disfavored flavor or for preferring the disfavored flavor. And we don’t approve of people for preferring the favored flavor.
Deep Blue has a goal (winning chess games), but I wouldn’t call that “morals”. OTOH, I can’t think of any decent explicit criterion for which goals I would call morals other than “I know it when I see it” at the moment.
A rough heuristic I use is that my moral preferences are those that I prefer regardless of whether I perceive the outcome (for example, deluding myself that other people aren’t suffering when they in fact are isn’t good), and my hedonic preferences are those where I only care about what I perceive (if I think what I’m eating tastes good, it doesn’t matter what it “really” tastes like).
This heuristic works for things like Deep Blue (it doesn’t care about chess games that it’s not aware of), but it doesn’t match my intuition for paperclippers. Any thoughts on why this heuristic breaks down there? Or is paperclipping simply a moral preference that I disapprove of, along the same lines as keeping women veiled or not eating shrimp?
I think that both morality, and the desires of paperclippers, are examples of what might be called “Non-personal preferences,” that is, they are preferences that, as you said, are preferred regardless of whether or not one perceives their fulfillment. All moral preferences are non-personal preferences, but not all Non-personal preferences are moral preferences.
The reason the heuristic works most of the time for you is, I think, because humans don’t have a lot of non-personal preferences. Having experiences is the main thing we care about. Morality is one of the few non-personal preferences we have. So if you have a preference you prefer regardless of whether or not you perceive the outcome, it is probably a moral preference.
The reason that heuristic breaks down in regards to paperclippers is that they are a hypothetical alien entity that has nothing but non-personal preferences. They aren’t human, their non-personal preferences aren’t moral.
What would I propose as a replacement heuristic? It’s a hard question, but I’d say moral preferences tend to have the following properties:
They are non-personal.
They are concerned about the wellbeing of people.
They are concerned about what sorts of people we ought to create.
They are usually fair and impartial in some (but not necessarily all) ways.
If you want an example of what might be a non-moral, non-personal preference that humans do have, I think a parent’s love for their children might be a candidate. Parents are willing to sacrifice large amounts of hedonic utility for their children even if they do not perceive the outcome of that sacrifice. And you can’t consider it a purely moral preference because the amount they are willing to sacrifice goes way beyond what a stranger would be morally obliged to sacrifice. If they sacrificed a stranger’s hedonic utility as freely as they sacrifice their own they would be justly condemned for nepotism.
This statement strikes me as semantic noise.
Joe should X. Jill should Y. Should as a relation between a being, it’s morality, and the actions it might take. That makes sense.
Who is the being with a morality in “there should exist”? The Universe? Non existent beings who should exist themselves into existence?
That was even worse, so I took the “that is” elaboration. But really, morality has a moral obligation to exist? Noise.
What the statement cashes out as, in more naturalistic phrasing, is “Creatures who know what is moral and act on it should take action to make sure that creatures like themselves continue to exist in the future and do not die out.”
I’m not sure that holds for all moralities, but it seems likely for consequentialist moralities.
If someone is there to fight for a set of values, it’s more likely those values are achieved. I’d think this extends more generally to power through existence all of time and space—a consequentialist will be motivated to increase the power of other consequentialists (for the same consequences), and likely decrease the power of conflicting moralities.
I assume the measure of “like” is similarity of morality, so that Clippy should make sure that beings with Clippy-like morality continue to exist.
This is certainly true, and one good reason to create moral beings. But I am also arguing that the creation of moral beings is a terminal goal of morality (providing, of course, that those moral beings live worthwhile lives).
I think we are using somewhat different terminology. You seem to be using the word “morality” as a synonym for “values” or “utility function.” I tend to follow Eliezer’s lead and consider morality to be a sort of abstract concept that concerns the wellbeing of people and what sort of creatures should be created. This set of concepts is one that most humans care about, and strongly desire to promote, but other creatures may care about totally different concepts.
So I would not consider paperclip maximizers, babyeaters, sociopaths, to be creatures that have any type of morality. They care about completely different concepts than the typical human being does, and referring to the things they care about as “morality” is confusing the issue.
Often I do, particularly when talking about moralities other than human.
Other times I will make what I consider a relevant functional distinction between general values and moral values in humans—moral values involves approval/disapproval in a recursive way—punishing an attitude, punishing failure to punish an attitude, punishing a failure to punish the failure to punish an attitude. etc. And similarly for rewards.
Elaborating on your point about supporting the continued existence of those who support your morality, on purely consequentialist grounds, Clippy is likely to engage in this kind of recursive punishment/reward as well, so he wouldn’t be entirely a Lovecraftian horror until the power of humans became inconsequential to his ends.
Likely Clippy will be “concerned” about the well being of people to the extent that such well being concerns the making of paperclips. Does that make him moral, according to your usage?
What kind of concern counts, and for which people must it apply? Concern is a very broad concept, with the alternative being unconcerned or indifferent. Clippy isn’t likely to be indifferent, and neither are sociopaths or baby eaters. Sociopaths and babyeaters are likely concerned about their own wellbeing, at least.
I don’t deny that it may be useful to create some sort of creature with nonhuman values for instrumental reasons. We do that all the time now. For instance, we create domestic cats, which if they are neurologically advanced enough to have values at all, are probably rather sociopathic; because of their instrumental value as companions and mousers.
Sorry, I should have been clearer. In order to count as morality the concern for the wellbeing of others has to be a terminal value. In Clippy’s case his concern is only an instrumental value, so he isn’t moral.
The speaker.
I’d unpack that sentence as “according to , a world with creatures who know what is is better than one without such creatures, all other things being equal”. AFAICT, that is correct when spoken by a non-psychopathic human (as human::morality does terminally value the existence of human::moral creatures), but not by a Pebblesorter (as, AFAICT, Pebblesorter::morality doesn’t care how many Pebblesorter::moral beings there are, except insofar as such beings will create more prime-numbered heaps).
Morality is an algorithm, so it exists the way infinitely many prime numbers exist, not the way I exist. So such a statement doesn’t even make sense—unless you take “morality” to be used synecdochally to refer to concrete implementations of the algorithm rather than to the abstract algorithm itself, in which case it reduces to the previous one.
The simpler and more accurate ways of expressing that are “Joe prefers A over B”, or to more clearly indicate the preference as a moral preference, “A is preferable to B according to Joe’s moral values”.
Should goes beyond a statement of preference or higher rank according to a set of values, and implies that someone has the moral obligation to make it happen, or more precisely, to take a certain action. According to my values, I’d prefer that I had magical powers, but there’s no person I can identify who has the moral obligation to give them to me, not even myself.
Why should morality exist? What do you mean by exist? I feel like there may have been a post or two before this one that I missed...
It’s just another way of saying that “it is good that creatures who know what is moral and act on it exist, and such creatures should make sure creatures like themselves continue to exist in the future.”
To make an analogy, if I said “Math should exist,” what I would actually mean is “Creatures that value doing math problems and studying mathematics should exist.”
This essay seems to me to badly go wrong, and it’s main error is begging all sorts of empirical questions against our maximizing utilitarian. (This seems to be the opposite mistake you made to the mere cable channel addition paradox, where you used empirical concerns to try and dodge the Repugnant conclusion).
1) The GC presumes that the best transfer of resources to utility will involve paperclips/animals, etc
(And so on with the sociopath and asteroid example).
You make the same mistake each time. You point to a population which you stipulate as having a higher total value than a population of normal humans, and then you assert that this particular human population should therefore act to have more paperclips/nonsapient animals/psychopaths. But this doesn’t follow: our maximizing utilitarian is obliged to accept that (for example) a certain number of happy mice is ‘worth the same’ as a happy human, and so a world with one less human but (say) a trillion more happy mice is better, but that does not necessarily recommend replacing humans with mice is a utilitarian good deal. It all depends on whether investing a given unit of resources into mice or people gives a higher utilitarian return.
Not only does nothing in the examples motivate thinking this will be the case, but there seem good a priori reasons to reject them: I reckon the ratio between mice and people is of the order to 100 000 : 1, so I doubt mice are a better utilitarian buy than persons, and similar concerns apply to paperclippers and sociopaths. Similarly, most utils will rate things like relationships, art etc. as worth a big chunk of ‘pleasure’ too, and so the assumption these things are better rendered down into being high or whatever else doesn’t seem plausible.
(Of course, empirically one can say the RC is not likely to be the resource-optimal utilitarian spend—there will be some trade off between making new people versus making the existing ones happier. But as covered in previous discussions is that what makes the RC repugnant is that it seems wrong to prefer such a world whether or not it is feasible. In the same way you might argue miceworld, clipperworld, or sociopathworld is repugnant to prefer to human world even if it isn’t feasible, but this recapitulates the standard RC. The GC wants to make this have more bite by saying utilitarianism should motivate us to create one of these scenarios, but it doesn’t work.)
2) Complexifying values won’t dodge the RC
We can specify all sorts of other values for an ideal consequentialism, but this won’t dodge the RC, and plausibly have its own problems. If the values are commensurable (i.e. we can answer ’how much is X increment of wisdom worth in terms of mutual affection), we basically get utilitarianism, as utils will likely say util is the common currency we use to cash out the value of these things. If they aren’t, we get moral dilemmas and instances where we cannot decide which is better (e.g., shall we make a person with slightly more wisdom, or slighlty more mutual affection?).
Even if we did, we can just stipulate an RC to interact with it. “Consider a population A with 10 increments of each feature of value (love, health, happiness, whatever), now A+, which has all members originally in A with 11 increments of each feature of value, and another population with 2 units of each feature of value, and now consider B, with all members in A+ at 9 units of each feature of value.” We seem to have just increased the dimensions of the problem.
We could have ideals about how utility is concentrated, and so have pattern goods that rule out RC like instances, but it seems horribly ad hoc to put ‘avoid repugnant conclusions’ as lexically prior in normative theory. It also seems a bit ad hoc or patch like to say “maximize utility in same number cases, but we need to do something different for different number cases”.
3) The case for rejecting mere (or beneficial) addition is weak
The case against the mere addition paradox is that people in fact do not have lots of children, and further that they don’t have lots of children and neglect them. I’m not adverse to taking folk opinion as evidence against ethical principles, but in this case the evidence is really weak. Maybe people are being ill-informed, or selfish, or maybe wary of the huge externalities of neglected children, or maybe they think (rightly or wrongly) that they add more utility directing their energy to other things besides having children.
It’s certainly true that total view util implies a pro tanto reason to have children, and even a pro tanto reason to have children even if you’ll look after them poorly. But you need way more work to show it generally secures an all things considered reason to have children, and waaaayyy more work to show it gives an all things considered reason to have children and neglect them.
#
Now hey, util has all sorts of counter-intuitive problems:
1) The fact value is fungible between love, mutual affection, getting high, feeling smug, having an orgasm, eating food, or reading Tolstoy means we get seemingly sucky cases where we should prefer wireheading or orgasmatrons to eudaimonia (now, utils can say eudaimonia is worth much more than wire-heading, but it won’t be incommensurately more, so enough wireheads will beat it).
2) That value is seperable between persons means we can get utility monsters, and cases where we benefit the best off instead of the worst off (or even benefit the best off at the expense of the worst off).
3) Aggregating leads to RCs and other worries.
And others I’ve forgotten. But I don’t think these are lethal: that utils are obliged in toy scenarios to genocide moral agents or have trillions of bare-worth-living lives doesn’t seem that embarassing to me—and, more importantly, alternative accounts of theories also run into problems too.
Ultimately, I don’t think your essay accomplishes more than tangling up the ‘usual suspects’ for anti total-utilitarianism objections, and the GC is a failure. I don’t think your program of ‘idealizing’ consequentialism in different number cases by adding on extra axes of value, or just bolting on ‘must not end up in RC’ to your normative theory is going to work well.
There does seem to be some good reason to consider mice a better buy, you can create a vast amount of happy mice for the same price you can create one human with all their various complex preferences. Mice require much fewer resources to give them pleasure than humans do to satisfy their complicated preferences.
You could rate these things as more valuable than the pleasure of animals, but that seems to go against the common utilitarian belief that the wellbeing of animals is important and that humans should be willing to sacrifice some of their wellbeing to benefit animals. After all, if rating those human values much better than animal pleasure means it’s better to create humans than animals, it also means it’s acceptable to inflict large harms on animals to help humans achieve relatively small gains in those values. I think it makes much more sense to say that we should value humans and animals equally if both exist, but prefer to create humans.
Not necessarily. We can stipulate that certain values are discontinuous, or have diminishing returns relative to each other. For instance, at the beginning I stipulated that “A population in which moral beings exist and have net positive utility, and in which all other creatures in existence also have net positive utility, is always better than a population where moral beings do not exist.” In other words a small village full of human beings living worthwhile lives is more worth creating than a galaxy full of mice on heroin.
Doing this doesn’t mean that you can’t aggregate the values in order to determine which ones it is best to promote. But it will prevent a huge quantity of one value from ever being able to completely dominate all the others.
I don’t think it’s an ad hoc theory just to avoid the RC. People seem to have similar moral views in a number of other cases that don’t involve vast populations. For instance, it’s generally agreed that it is wrong to kill someone who is destined to live a perfectly good and worthwhile life even if doing so will allow you to replace them with another person who will live a slightly better life. It seems like “maximize utility” is a principle that only applies to existing people. Creating more people requires a whole other set of rules.
The more I think about it, the more I have trouble believing I ever thought utility maximization was a good way to do population ethics. It seems like an instance of “when all you have is a hammer everything looks like a nail.” Utility maximization worked so well for scenarios where the population was unchanged that ethicists tried to apply it to population ethics, even though it doesn’t really work well there.
I personally don’t find the classical RC as repugnant as I used to. The implications of total utilitarianism that I still find deal-breakingly abhorrent are conclusions that involve killing a high utility population in order to replace it with a slightly higher utility population, especially in cases where the only reason the new population has higher utility is that its members have simpler, easier to satisfy values. The Genocidal Conclusion is the big one, but I also find micro-level versions of these conclusions horrible, like that it might be good to wirehead someone against their will, or kill someone destined to lead a great life and replace them with someone with a slightly better life. It was these conclusions that led me to ideal consequentialism, the fact that it might also be able to avoid the RC was just icing on the cake. If ideal consequentialism can somehow avoid “kill and replace” conclusions, but can’t avoid the RC, I’d still be satisfied with it.
I don’t think it’s any weirder than saying “maximize preference satisfaction for creatures capable of having preferences, but maximize pleasure for creatures that are not.” And that doesn’t seem a controversial statement at all.
I agree that that is the weakest part of the OP. But I don’t think people are motivated by the reasons you suggested. Most people I know seem to think that they have a moral duty to provide a high level of care to any children they might have, and have a moral duty to not have children if they are unwilling or unable to provide that level of care. Their motivation is driven by moral conviction, not by selfishness or by practical considerations.
In general people’s preference ordering seems to be:
1) New person exists and I am able to care for them without inflicting large harms on myself.
2) New person doesn’t exist
3) New person exists and I inflict large harms on myself to care for them.
4) New person exists and I don’t care for them.
If this is the case then it might be that mere or benign addition is logically impossible. You can’t add new lives barely worth living without harming the interests of existing people.
That does seem to be so, but why? I’d change the order of 1) and 2). What’s so good about having more people? Better improve the lives of existing people instead.
My theory at the moment is that we have some sort of value that might be called “Harmony of self-interest and moral interests.” This value motivates us to try to make sure the world we live in is one where we do not have to make large sacrifices of our own self-interest in order to improve the lives of others. This in turn causes us to oppose the creation of new people with lives that are worse than our own, even if we could, in theory, maintain our current standard of living while allowing them to live in poverty. This neatly blocks the mere addition paradox since it makes it impossible to perform “mere additions” of new people without harming the interests of those who exist.
I suspect the reason this theory is not addressed heavily in moral literature is the tendency to conflate utility with “happiness.” Since it obviously is possible to “merely add” a new person without impacting the happiness of existing people (for instance, you could conceal the new person’s existence from others so they won’t feel sorry for them) it is mistakenly believed you can also do so without affecting their utility. But even brief introspection reveals that happiness and utility are not identical. If someone spread dirty rumors about me behind my back, cheated on me, or harmed my family when I wasn’t around, and I never found out, my happiness would remain the same. But I’d still have been harmed.
You’re right, of course. But at this point in our history it isn’t actually a choice of one or the other. Gradually adding reasonable amounts of people to the world actually improves the lives of existing people. This is because doing so allows the economy to develop new divisions of labor that increase the total amount of wealth. That is why we are so much richer than we were in the Middle Ages, even though the population is larger. Plus humans are social animals, having more people means having more potential friends.
Eventually we may find some way to change this. For instance, if we invented a friendly AI that was cheaper to manufacture than a human, more efficient at working, and devoted all its efforts to improving existing lives, then mass manufacturing more copies of it would increase total wealth better than making more people (though we would still be justified in making more people if the relationships we formed with them greatly improved our lives).
But at the moment there is no serious dilemma between adding a reasonable amount of new people and improving existing lives. The main reason I am going after total utilitarianism so hard is that I like my moral theories to be completely satisfactory, and I disapprove of moral theories that give the wrong answers, even if they only do so in a scenario that I will never encounter in real life.
Step 1 -- Replace humanity with a sufficiently large number of paperclip satisficers (they want to make only a few very specific paperclips to be absolutely happy, but each of them wants a different kind of a paperclip). If you give a non-zero value to a paperclips satisficer, then for a sufficiently large number of paperclip satisficers, replacing humanity with them should be considered a good choice.
Step 2 -- Paperclip satisficers replace themselves with paperclips. (Because they are satisficers and not maximizers, they don’t care about expanding and transforming the whole universe.)
Game Over
Is the size of the cosmological event horizon until the heat death of the local area large enough to postulate enough paperclip satisficers? Since we are dealing with finite time and space, it is not a given that a sufficiently large number of anything would be possible, and we are discussing how large numbers are counterintuititve rather than a moral point.
I would argue that the solution to this may be to make the value of having humans, or some other type of moral being, exist, be discontinuous in some fashion.
In other words, adding paperclip satisficers might add some value, but in a world where no humans (or other type of moral being) exists there is no amount of paperclip satisficers you can add that would be as valuable as creating humans. Similarly, there is no amount of paperclip satisficers that can be added that could ever replace the human race.
Now please note that I am not trying to be speciesist when I talk about humans and human values. A creature that shares human values, but isn’t genetically a member of the human species, is far more worth creating than a creature that is genetically a member of the human species, but lacks certain values. For instance, an alien that had evolved human-like values through parallel evolution is far more worth creating than a human sociopath.
But what if mere paper clip satisficers simply aren’t capable of experiencing the full joy of being a great paperclip maximiser who has paperclipped the universe? I think the simplest solution is to measure joy in number of paperclips made =]
Paperclip satisficers, by definition, are completely happy with producing their paperclips, and absolutely don’t mind missing the joys of being a paperclip maximizer.
Essentially, it is a question of what happens first. If as a first step we replace all human beings with enough paperclip satisficers to increase the total utility, in the resulting world nobody will attempt to create a paperclip maximizer. If something else happens first, the results may be completely different.
There are two different ways that I sometimes think about population ethics, and I think it’s useful to keep these situations distinct.
First, suppose you are altruistic, and given the chance to make decisions affecting only people other than yourself. From this perspective, the mere addition paradox would involve redistributing utility among a population that I am not a member of, and I accept the repugnant conclusion without much reservation. I would even accept killing off the current population (that I am not a member of) if that allowed me to replace it with a population whose lives are barely worth living, if the size of the potential new population were sufficiently high. My response to your other modifications is that I am not altruistic towards paperclippers, nonsentients, etc, so I would not harm a human population for their benefit. This may seem unobjective, but that is necessary. There is no objective way to compare utilities between different agents, to say that gaining a dollar is 10 times as valuable to Alice as it is to Bob. The closest I can do is say that I care about Alice gaining a dollar 10 times as much as I care about Bob gaining a dollar.
The other perspective that I look at population ethics problems from is that of a typical member of the original population being affected. Here, creating an additional population whose lives are barely worth living, while not changing the utilities of the original population, is neutral. Creating an additional population whose lives are barely worth living while increasing the utilities of the original population (such an increase could happen due to altruism towards the newly created people) is good. And creating an additional population while decreasing the utilities of the original population (e.g. through resource redistribution) is a bad thing. Let’s suppose the original population is given the opportunity to create a new population, whose lives are barely worth living, and gain some utility themselves. Suppose they take this opportunity, and afterwards, the new, larger population, has the option to redistribute resources, increasing average utility, but decreasing the utilities of the original population below where they were originally. Locally, this looks like a good idea. But if redistribution occurs, then it would not have been a good idea for the original population to create the additional population in the first place, so by TDT, redistribution is a bad idea in that scenario, and the repugnant conclusion is averted.
No it isn’t.
I expected some people might reject the paperclipper example. That’s why I also included the ones with animals and sociopaths. What do you think of those?
Personally, I think the same about the animals, and am in doubt but lean towards thinking the same about the sociopaths. OTOH, people who think it would be a good thing for animals to be better off even if humans were worse off aren’t very rare, and I have definitely seen several people on Facebook argue (I’m not sure how seriously) that it would be a good thing for humans to extinct if that made animals better off.
How close is close enough? If my moral system has 20 distinct components, should I value a being with 19 of those 20 as a moral being? 95% of a moral being? What if it has 5 of those 20 components?
Assuming that all components are equally valuable (which probably isn’t true in real life), and that the whole isn’t greater than the sum of its parts (again, this is probably false in real life) then I think that being with “partial morality” may be “partially worth creating.”
This sounds much less strange if you state it in more concrete terms. If you ask people to rank the following scenarios in preference order:
All life on Earth goes extinct.
Humanity, and all other life on Earth, survives perfectly fine.
Humanity dies out, but the other great apes do not.
Most people would rank, in order from most to least desirable, 2, 3, 1. This is because the other great apes likely share at least some moral values in common with humanity, so their survival is better than nothing.
Something I have noted about Repugnant Conclusion arguments as they relate to Population ethics from reading this lead me to have a question about whether an idea I had was reasonable:
In Repugnant Conclusion arguments in general, you are losing Average Utility, but gaining Total Utility, with a rate of change that seems to vary heavily or remain unspecified from argument to argument. I think the rate of change of average and total utility seems to affect the overall repugnance of the chain of trades rather heavily.
(Note: I’m using towns since fractional towns makes more sense than fractional people)
Let’s say you begin with 1 towns of people at 10 average utility, Towns x Utility is 10.
and the next step is 1.222… towns of people at 9 average utility. Towns x Utility is 11.
With the next step, you might either:
A: Continue losing 1 average utility and gaining 1 total utility each step.
B: Continue losing 10% remaining average utility and gaining 10% total utility each step.
C: Continue raising average utility to approximately the 0.955 power with each step and raising Total utility to approximately the 1.041 power with each step.
A seems substantially more repugnant than B, and each step grows seemingly more repugnant in percentage terms. You trade a greater proportion of your average utility away on each trade for increasingly smaller percentage gains to total utility, so arguably at each point it looks like your next deal is getting worse. And in A you can’t actually get to 21 towns without going to negative total utility.
Whereas in B, you won’t ever actually reach 0 or negative average utility, even when they have towns> number of atoms in the universe, And at each point, you trade a equal proportion of your average utility away on each trade for an equal percentage gains to total utility, so arguably at each point it looks like your next deal is going to be of comparable quality to your previous deal. However, even in the case of B, the average utility is still approaching 0, which is usually considered crappy.
But with C, the average utility is instead approaching 1 and the total utility grows substantially faster. Each deal actually grows better over time in percentage terms. (At first, you trade away around 10% of your average utility to get around 10% more total utility, but later, you might be trading away only 2% of your remaining average utility to gain 50% more total utility. And at each point you would expect to get better trades in the future.)
I would hesitate to call B a repugnant solution when compared to how awful A is (Unless I call A the “Genocidal Conclusion” as per Ghatanathoah), and I would hesitate to call C a repugnant solution at all. Does that seem reasonable?
Edited to fix formatting
This seems obviously absurd. The examples you listed are all valuable things, but only because I (and others) value them.
I certainly don’t think that the things I listed are valuable independent of creatures that value them existing. For instance, I don’t think that a lifeless world full of beautiful statues or great books would be any more valuable than any other kind of lifeless world. And as I say later, trying to force a person who doesn’t value those things to do them is not valuable at all.
What I am saying is that creating creatures that value those things is morally good and attempting to replace creatures that value those things is morally bad. A village full of humans living worthwhile lives is morally better than a galaxy full of paperclip maximizers
I agree, but only because it would likely increase the utility of currently existing creatures, such as myself.
I disagree with your reasoning in this respect. It seems to me like I have some sort of moral duty to make sure creatures with certain values exist, and this is not only because of their utility, or the utility they bring to existing creatures.
For instance, if I had a choice between: A) Everyone on Earth dies in a disaster, later a species with humanlike values evolves or B) Everyone on Earth dies in a disaster, later a race of creatures with extremely unhumanlike values evolves, I would pick A, even though neither choice would increase the utility of anyone existing. I would even pick A if the total utility of the creatures in A was lower than the total utility of the creatures in B, as long as the total utility was positive.
I agree that ultimately, the only reason I am motivated to act in such a fashion is because of my desires and values. But that does not mean that morality reduces only to satisfying desires and values. It means that it is morally good to create such creatures, and I desire and value acting morally.
A is better than B for currently existing people.
As I pointed out in my other response to your post, that doesn’t actually mean anything
I would choose A even if it meant existing people would have to make some sort of small sacrifice before the disaster killed them.
And in the case of the the Repugnant Conclusion I might choose A or Q over Z even if every single person is created at the same time instead of one population existing prior to the other.
I’m not referring to VNM utility, what I’m talking about is closer to what the Wikipedia entry calls “E-utility.” To summarize the difference, if I do something I’ll be miserable doing because it greatly benefits someone else and that is morally good, I will have higher VNM utility than if I didn’t do it, because VNM utility includes doing what is morally good as part of my utility function. My E-utility, however, will be lower than if I didn’t do it.
Interpersonal E-utility comparisons aren’t that hard. I do them all the time in my day-to-day life. That’s basically what the human capacity for empathy is. I use my capacity for empathy to model other people’s minds. In fact, if it wasn’t possible to do interpersonal E-utility comparisons between agents it’s kind of hard to see how humans would have ever even evolved the capacity for empathy..
So would I. As evidenced by our choices, we care enough about A vs B that A is still better than B in terms of our (VNM) utility even if A requires us to make a small local sacrifice.
If you were talking about E-utility this entire time, then that changes everything. Our preferences over people’s E-utilities are tremendously complicated, so any ideal population ethics described in terms of E-utilities will also be tremendously complicated. It doesn’t help that E-utility is a pretty fuzzy concept.
Also, saying that you’re talking about E-utility doesn’t solve the problem of interpersonal utility comparison. Humans tend to have similar desires as other humans, so comparing their utility isn’t too hard in practice. But how would you compare the E-utilities experienced by a human and a paperclip maximizer?
I was, do you think I should make a note of that somewhere in the OP? I should have realized that on a site that talks about decision theory so often I might give the impression I was talking about VNM instead of E.
That is difficult, to be sure. Some kind of normalizing assumption is probably neccessary. One avenue of attack would be the concept of a “life worth living,” for a human it would be a life where positive experiences outweighed the negative, for a paperclipper one where its existence resulted in more paperclips than not.
It may be that the further we get away from the human psyches the fuzzier the comparisons get. I can tell that a paperclipper whose existence has resulted in the destruction of a thousand paperclips has a lower E-utility than a human who lives a life very much worth living. But I have trouble seeing how to determine how much E-utility a paperclipper who has made a positive number of paperclips has compared to that person.
That might be a good idea.
That tells you what zero utility is, but it doesn’t give you scale.
It sounds like you’re equating E-utility with VNM utility for paperclippers. It seems more intuitive to me to say that paperclippers don’t have E-utilities, because it isn’t their experiences that they care about.
I added a footnote.
That’s probably right. That also bring up what I consider an issue in describing the utility of humans. Right now we are basically dividing the VNM of humans into E-utility and what might be termed “moral utility” or “M-utility.” I’m wondering if there is anything else. That is, I wonder if human beings have any desires that are not either desires to have certain experiences, or desires to do something they believe is morally right. Maybe you could call it “Nonpersonal nonmoral utility,” or “NN utility for short.”
I wracked my brain and I can’t think of any desires I have that do not fit the categories of M or E utility. But maybe I’m not just thinking hard enough. Paperclippers are obviously nothing but NN utility, but I wonder if it’s present in humans at all.
Aesthetics? ETA: I seem to be able to coherently desire the existence of artwork that I will never see and the completion of processes I will not interact with.
Presumably other people will see and enjoy that artwork, so aesthetics in that case might be a form of morality (you care about other people enjoying art, even if you never see that art yourself).
On the other hand, if you desire the existence of an aesthetically beautiful* natural rock formation, even if it was on a lifeless planet that no one would ever visit, that might count as NN utility.
*Technically the rock formation wouldn’t be beautiful since something’s beauty is a property of the mind beholding it, not a property of the object. But then you could just steel man that statement to say you desire the existence of a rock formation that would be found beautiful by a mind beholding it, even if no mind ever beholds it.
Actually, my aesthetics seem to be based as much on very powerful and possibly baseless intuitions of “completeness” as they do about beauty.
Omega offers to create a dead universe containing a printout of all possible chess games, and a printout of all possible chess games minus n randomly selected ones; my absurdly strong preference for the former is unaffected by his stipulation that no agent will ever interact with said universe and that I myself will immediately forget the conversation. And I don’t even like chess.
What’s the difference between M-utility and NN-utility? Does it have to do with the psychology of why you have the preference? What if there’s an alien with radically different psychology from us, and they developed morality-like preferences to help them cooperate, but they don’t think about them the way we think about morality? Would they have M-utility? Also, the separation between E-utility and M/NN-utility will get fuzzy if we can make uploads.
It’s hard to explicitly define the difference. I feel like preferring that other people have positive utility, and preferring that human beings exist instead of paperclippers, is a moral preference, whereas preferring that paperclips exist is an NN preference. So maybe M-utility involves my nonpersonal preferences that are about other people in some way, while NN preferences are nonpersonal preferences about things other than people.