“Utilitarianism” has two different, but related meanings. Historically, it generally means
“the morally right action is the action that produces the most good”, or as Bentham put it, “the greatest amount of good for the greatest number”. Leave aside for the moment that this ignores the tradeoff between how much good and how many people, and exactly what the good is. Bentham and like-minded thinkers mean by “good” things like material well-being, flourishing, “happiness”, and so on. They are pointing in a certain direction, even if a bit vaguely. Utilitarianism in this sense is about people, and its conception of the good consists of what humans generally want. It is necessarily expressed in terms of human concepts, because that is what it is about.
The other thing that the word “utilitarianism” has become used for is the thing that various theorems prove can be constructed from a preference relation satisfying certain axioms. Von Neumann and Morgenstern are the usual names mentioned, but there are also Savage, Cox, and others. Collectively, these are, as Eliezer has put it, “multiple spotlights all shining on the same core mathematical structure”. The theory is independent of of any specific preference relation and of what the utility function determined by those preferences comes out to be. (ETA: This use of the word might be specific to the rationalist community. “Utility theory” is I think the more widely used term. Accordingly I’ve replaced “VNMU” by VNMUT” below.)
To distinguish these two concepts I shall call them “Benthamite utilitarianism” and “Von Neuman-Morgenstern utility theory”, or BU and VNMUT for short. How do they relate to each other, and what does either have to say about AI?
BU has a specific notion of the individual good. VNMUT does not. VNMUT is concerned only with the structure of the preference relation, not its content. In VNMUT, the preference relation is anything satisfying the axioms; in BU it is a specific thing, not up for grabs, described by words such as “welfare”, “happiness”, or “satisfaction”.
By analogy: BU is like studying the structure of some particular group, such as the Monster Group, while VNMUT is like group theory, which studies all groups and does not care where they came from or what they are used for.
VNMUT is made of theorems. BU is not. BU contains no mathematical structure to elucidate what is meant by “the greatest good for the greatest number”. The slogan is a rallying call, but leaves many hard decisions to be made.
Neither BU nor VNMUT have a satisfactory concept of collective good. BU is silent about the tradeoff between the greatest good and the greatest number. There is no generally agreed on extension of VNMUT to mathematically construct a collective preference relation or utility function. There have been many attempts, on both the practical side (BU) and the theoretical side (VNMUT), but the body of such work does not have the coherence of those “multiple spotlights all shining on the same core mathematical structure”. The differing attitudes we observe to the Repugnant Conclusion illustrate the lack of consensus.
What do either of these have to do with AI?
If a program is trained to produce outputs that maximise some objective function, that value is at least similar to a utility in the VNMUT sense, although it is not derived from a preference relation. The utility (objective function) is primitive and a preference relation can be derived from it: the program “prefers” a higher value to a lower.
As for BU, whether a program optimises for the human good is up to what its designers choose to have it optimise. Optimise for deadly poisons and that may be what you get. (I don’t know if anyone has experimented with the compounds that that experiment suggested, although it seems to me quite likely that some military lab somewhere is doing so, if they weren’t already.) Optimise for peace and love, and maybe you get something like that, or maybe you end up painting smiley faces onto everything. The AI itself is not feeling or emoting. Its concepts of “welfare”, “happiness”, or “satisfaction”, such as they are, are embodied in the training procedure its programmers used to judge its outputs as desired or undesired.
“Conclusively” is doing too much work there. Do you attribute feelings or emotions to current AIs? I deny them on the same grounds as I deny them to any of the other software I use, and to rocks. I say current AIs, because that is what I had in mind, and because there would be no point in arguing “But suppose someone did make an AI with emotions! Then it would have emotions!”
I just gave my answer. For more, there’s this from my recent ding-dong with Signer. Briefly, in the absence of any method of actually detecting and measuring consciousness (a concept in which I include feelings and emotions), a consciousnessometer, we must fall back on the experiences that give rise to the very concept, on the basis of which we attribute consciousness to people besides ourselves, and to some extent other animals. On that basis I see no reason to attribute it to any extant piece of software.
Briefly, in the absence of any method of actually detecting and measuring consciousness (a concept in which I include feelings and emotions), a consciousnessometer, we must fall back on the experiences that give rise to the very concept, on the basis of which we attribute consciousness to people besides ourselves, and to some extent other animals.
That seems like a less popular understanding.
Why must consciousness include ‘feelings’ and ‘emotions’?
If someone has their portion of the brain responsible for emotional processing damaged, do they become less conscious?
Merriam-webster also lists that as number 2 in their dictionary, and a different definition in the number one position:
Definition of consciousness
1a: the quality or state of being aware especially of something within oneself
b: the state or fact of being conscious of an external object, state, or fact
c: AWARENESSespecially: concern for some social or political cause The organization aims to raise the political consciousness of teenagers.
2: the state of being characterized by sensation, emotion, volition, and thought : MIND
3: the totality of conscious states of an individual
4: the normal state of conscious liferegained consciousness
5: the upper level of mental life of which the person is aware as contrasted with unconscious processes
Why must consciousness include ‘feelings’ and ‘emotions’?
If they are present, they are part of consciousness. They are included in the things of which one is aware within oneself (and in item 2 of the definition you quote). I did not intend any implication that they must be present, for consciousness to be present.
“Utilitarianism” has two different, but related meanings. Historically, it generally means “the morally right action is the action that produces the most good”, or as Bentham put it, “the greatest amount of good for the greatest number”. Leave aside for the moment that this ignores the tradeoff between how much good and how many people, and exactly what the good is. Bentham and like-minded thinkers mean by “good” things like material well-being, flourishing, “happiness”, and so on. They are pointing in a certain direction, even if a bit vaguely. Utilitarianism in this sense is about people, and its conception of the good consists of what humans generally want. It is necessarily expressed in terms of human concepts, because that is what it is about.
The other thing that the word “utilitarianism” has become used for is the thing that various theorems prove can be constructed from a preference relation satisfying certain axioms. Von Neumann and Morgenstern are the usual names mentioned, but there are also Savage, Cox, and others. Collectively, these are, as Eliezer has put it, “multiple spotlights all shining on the same core mathematical structure”. The theory is independent of of any specific preference relation and of what the utility function determined by those preferences comes out to be. (ETA: This use of the word might be specific to the rationalist community. “Utility theory” is I think the more widely used term. Accordingly I’ve replaced “VNMU” by VNMUT” below.)
To distinguish these two concepts I shall call them “Benthamite utilitarianism” and “Von Neuman-Morgenstern utility theory”, or BU and VNMUT for short. How do they relate to each other, and what does either have to say about AI?
BU has a specific notion of the individual good. VNMUT does not. VNMUT is concerned only with the structure of the preference relation, not its content. In VNMUT, the preference relation is anything satisfying the axioms; in BU it is a specific thing, not up for grabs, described by words such as “welfare”, “happiness”, or “satisfaction”.
By analogy: BU is like studying the structure of some particular group, such as the Monster Group, while VNMUT is like group theory, which studies all groups and does not care where they came from or what they are used for.
VNMUT is made of theorems. BU is not. BU contains no mathematical structure to elucidate what is meant by “the greatest good for the greatest number”. The slogan is a rallying call, but leaves many hard decisions to be made.
Neither BU nor VNMUT have a satisfactory concept of collective good. BU is silent about the tradeoff between the greatest good and the greatest number. There is no generally agreed on extension of VNMUT to mathematically construct a collective preference relation or utility function. There have been many attempts, on both the practical side (BU) and the theoretical side (VNMUT), but the body of such work does not have the coherence of those “multiple spotlights all shining on the same core mathematical structure”. The differing attitudes we observe to the Repugnant Conclusion illustrate the lack of consensus.
What do either of these have to do with AI?
If a program is trained to produce outputs that maximise some objective function, that value is at least similar to a utility in the VNMUT sense, although it is not derived from a preference relation. The utility (objective function) is primitive and a preference relation can be derived from it: the program “prefers” a higher value to a lower.
As for BU, whether a program optimises for the human good is up to what its designers choose to have it optimise. Optimise for deadly poisons and that may be what you get. (I don’t know if anyone has experimented with the compounds that that experiment suggested, although it seems to me quite likely that some military lab somewhere is doing so, if they weren’t already.) Optimise for peace and love, and maybe you get something like that, or maybe you end up painting smiley faces onto everything. The AI itself is not feeling or emoting. Its concepts of “welfare”, “happiness”, or “satisfaction”, such as they are, are embodied in the training procedure its programmers used to judge its outputs as desired or undesired.
How can we know conclusively that ‘The AI itself is not feeling or emoting.’?
“Conclusively” is doing too much work there. Do you attribute feelings or emotions to current AIs? I deny them on the same grounds as I deny them to any of the other software I use, and to rocks. I say current AIs, because that is what I had in mind, and because there would be no point in arguing “But suppose someone did make an AI with emotions! Then it would have emotions!”
If the objectionable word is removed:
Would you have a different answer?
I just gave my answer. For more, there’s this from my recent ding-dong with Signer. Briefly, in the absence of any method of actually detecting and measuring consciousness (a concept in which I include feelings and emotions), a consciousnessometer, we must fall back on the experiences that give rise to the very concept, on the basis of which we attribute consciousness to people besides ourselves, and to some extent other animals. On that basis I see no reason to attribute it to any extant piece of software.
That seems like a less popular understanding.
Why must consciousness include ‘feelings’ and ‘emotions’?
If someone has their portion of the brain responsible for emotional processing damaged, do they become less conscious?
Merriam-webster also lists that as number 2 in their dictionary, and a different definition in the number one position:
If they are present, they are part of consciousness. They are included in the things of which one is aware within oneself (and in item 2 of the definition you quote). I did not intend any implication that they must be present, for consciousness to be present.