I’d appreciate clarification on what you mean by “You should_me maximize my preferences.”
I understand that the “objective” part is that we could both come to agree on the value of should_you and the value of should_me, but what do you mean when you say that I should_MattSimpson maximize your preferences?
I certainly balk at the suggestion that there is a should_human, but I’d need to understand Eliezer in more detail on that point.
And yes, if one’s own preferences are the foundation of ethics, most philosophers would simply call this subject matter practical rationality rather than morality. “Morality” is usually thought to be a term that refers to norms with a broader foundation and perhaps even “universal bindingness” or something. On this point, Eliezer just has an unusual way of carving up concept space that will confuse many people. (And this is coming from someone who rejects the standard analytic process of “conceptual analysis”, and is quite open to redefining terms to make them more useful and match the world more cleanly.)
Also, even if you think that the only reasons for action that exist come from relations between preferences and states of affairs, there are still ways to see morality as a system of hypothetical imperatives that is “broader” (and therefore may fit common use of the term “morality” better) than Eliezer’s meta-ethical theory. See for example Peter Railton or 1980s Philippa Foot or, well, Alonzo Fyfe and Luke Muehlhauser.
We already have a term that matches Eliezer’s use of “ought” and “should” quite nicely: it’s called the “prudential ought.” The term “moral ought” is usually applied to a different location in concept space, whether or not it successfully refers.
Anyway, are my remarks connecting with Eliezer’s actual stated position, do you think?
but what do you mean when you say that I should_MattSimpson maximize your preferences?
I mean that according to my preferences, you, me, and everyone else should maximize them. If you ask what should_MattSimpson be done, the short answer is maximize my preferences. Similarly, if you ask what should_lukeproq be done, the short answer is to maximize your preferences. It doesn’t matter who does the asking. If you ask should_agent should be done, you should maximize agent’s preferences. There is no “should” only should_agent’s. (Note, Eliezer calls should_human “should.” I think it’s an error of terminology, personally. It obscures his position somewhat).
We already have a term that matches Eliezer’s use of “ought” and “should” quite nicely: it’s called the “prudential ought.” The term “moral ought” is usually applied to a different location in concept space, whether or not it successfully refers.
Then Eliezer’s position is that all normativity is prudential normativity. But without the pop-culture connotations that come with this position. In other words, this doesn’t mean you can “do whatever you want.” You probably do, in fact, value other people, you’re a human after all. So murdering them is not ok, even if you know you can get away with it. (Note that this last conclusion might be salvageable even if there is no should_human.)
As for why Eliezer (and others here) think there is a should_human (or that human values are similar enough to talk about such a thing), the essence of the argument rests on ev-psych, but I don’t know the details beyond “ev-psych suggests that our minds would be very similar.”
Does Eliezer claim that murder is wrong for every agent? I find it highly likely that in certain cases, an agent’s murder of some person will best satisfy that agent’s preferences.
Murder is certainly not wrong_x for every agent x—we can think of an agent with a preference for people being murdered, even itself. However, it is almost always wrong_MattSimpson and (hopefully!) almost always wrong_lukeproq. So it depends on which question your are asking. If you’re asking “is murder wrong_human for every agent?” Eliezer would say yes. If you’re asking “is murder wrong_x for every agent x?” Eliezer would say no.
(I realize it was clear to both you and me which of the two you were asking, but for the benefit of confused readers, I made sure everything was clear)
I would be very surprised if EY gave those answers to those questions.
It seems pretty fundamental to his view of morality that asking about “wrong_human” and “wrong_x” is an important mis-step.
Maybe murder isn’t always wrong, but it certainly doesn’t depend (on EY’s view, as I understand it) on the existence of an agent with a preference for people being murdered (or the absence of such an agent).
Maybe murder isn’t always wrong, but it certainly doesn’t depend (on EY’s view, as I understand it) on the existence of an agent with a preference for people being murdered (or the absence of such an agent).
That’s because for EY, “wrong” and “wrong_\human” mean the same thing. It’s semantics. When you ask “is X right or wrong?” in the every day sense of the term, you are actually asking “is X right_human or wrong_human?” But if murder is wrong_human, that doesn’t mean it’s wrong_clippy, for example. In both cases you are just checking a utility function, but different utility functions give different answers.
It seems clear from the metaethics posts is that if a powerful alien race comes along and converts humanity into paperclip-maximizers, such that making many paperclips comes to be right_human, EY would say that making many paperclips doesn’t therefore become right.
So it seems clear that at least under some circumstances, “wrong” and “wrong_human” don’t mean the same thing for EY, and that at least sometimes EY would say that “is X right or wrong?” doesn’t depend on what humans happen to want that day.
Now, if by “wrong_human” you don’t mean what humans would consider wrong the day you evaluate it, but rather what is considered wrong by humans today, then all of that is irrelevant to your claim.
In that case, yes, maybe you’re right that what you mean by “wrong_human” is also what EY means by “wrong.” But I still wouldn’t expect him to endorse the idea that what’s wrong or right depends in any way on what agents happen to prefer.
It seems clear from the metaethics posts is that if a powerful alien race comes along and converts humanity into paperclip-maximizers, such that making many paperclips comes to be right_human
No one can change right_human, it’s a specific utility function. You can change the utility function that humans implement, but you can’t change right_human. That would be like changing e^x or 2 to something else. In other words, you’re right about what the metaethics posts say, and that’s what I’m saying too.
edit: or what jimrandomh said (I didn’t see his comment before I posted mine)
What if we use ‘human’ as a rigid designator for unmodified-human. Then in case aliens convert people into paperclip-maximizers, they’re no longer human, hence human_right no longer applies to them, but itself remains unchanged.
human_right still applies to them in the sense that they still should do what’s human_right. That’s the definition of should. (Remember, should refers to a specific set of terminal values, those that humans happen to have, called human_right) However, these modified humans, much like clippy, don’t care about human_right and so won’t be motivated to act based on human_right (except insofar as it helps make paperclips).
I’m not necessarily disagreeing with you because it’s a little ambiguous how you used the word “applies.” If you mean that the modified humans don’t care about human_right anymore, I agree. If you mean that the modified humans shouldn’t care about human_right, then I disagree.
I’m not sure why it’s necessary to use ‘should’ to mean morally_should, it could just be used to mean decision-theoretic_should. E.g. if you’re asked what a chess playing computer program should do to win a particular game, you could give a list of moves it should make. And when a human asks what they should do related to a moral question, you can first use the human_right function to determine what is the desired state of the world that they want to achieve, and then ask what you should do (as in decision-theoretic_should, or as in what moves/steps you need to execute, in analogy to the chess program) to create this state. Thus morality is contained within the human_right function and there’s no confusion over the meaning of ‘should’.
As long as you can keep the terms straight, sure. EY’s argument was that using “should” in that sense makes it easier to make mistakes related to relativism.
OK. At this point I must admit I’ve lost track of why these various suggestively named utility functions are of any genuine interest, so I should probably leave it there. Thanks for clarifying.
It seems clear from the metaethics posts is that if a powerful alien race comes along and converts humanity into paperclip-maximizers, such that making many paperclips comes to be right_human, EY would say that making many paperclips doesn’t therefore become right.
In that case, we would draw a distinction between right_unmodified_human and right_modified_human, and “right” would refer to the former.
If you ask what should_MattSimpson be done, the short answer is maximize my preferences.
I find the talk of “should_MattSimpson” very unpersuasive given the availability of alternative phrasings such as “approved_MattSimpson” or “valued_MattSimpson”. I have read below that EY discourages such talk, but it seems that’s for different reasons than mine. Could someone please point me to at least one post in the sequence which (almost/kinda/sorta) motivates such phrasings?
Alternate phrasings such as those you listed would probably be less confusing, i.e. replacing “should” in “should_X” with “valued” and reserving “should” for “valued_human”.
And yes, if one’s own preferences are the foundation of ethics, most philosophers would simply call this subject matter practical rationality rather than morality.
They would be missing some important distinctions between what we think of as our moral values and what we think of as “chocolate/vanilla” preferences. For one obvious example, consider an alien ray gun that ‘switches the way I feel’ about two things, X and Y, without otherwise affecting my utility function or anything else of value to me.
If X were, say, licorice jelly beans (yum) and Y were, say, buttered popcorn jelly beans (yuck), then I wouldn’t be too deeply bothered by the prospect of being zapped with this gun. (Same for sexual preference, etc.) But if X were “autonomy of individuals” and Y were “uniformity of individuals”, I would flee screaming from the prospect of being messed with that way, and would take some extreme actions (if I knew I’d be zapped) to prevent my new preferences from having large effects in the world.
Now we can develop whole theories about what this kind of difference consists in, but it’s at least relevant to the question of metaethics. In fact, I think that calling this wider class of volitions “preferences” is sneaking in an unfortunate connotation that they “shouldn’t really matter then”.
This sounds, to me, like it’s just the distinction between terminal and instrumental values. I don’t terminally value eating licorice jelly beans, I just like the way they taste and the feeling of pleasure they give me. If you switched the tastes of buttered popcorn jelly beans (yuck indeed) and licorice jelly beans, that would be fine by me. Hell, it would be an improvement since no one else likes that flavor (more for me!). The situation is NOT the same for “autonomy of individuals” and “uniformity of individuals” before I really do have terminal values for these things, apart from the way they make me feel.
The situation is NOT the same for “autonomy of individuals” and “uniformity of individuals” before I really do have terminal values for these things, apart from the way they make me feel.
How do you know that?
What would you expect to experience if your preference for individual autonomy in fact derived from something else?
I agree that by using a single term for the wider class of volitions—for example, by saying both that I “prefer” autonomy to uniformity and also that I “prefer” male sexual partners to female ones and also that I “prefer” chocolate to vanilla—I introduce the connotation that the distinctions between these various “preferences” aren’t important in the context of discourse.
To call that an unfortunate connotation is question-begging. Sometimes we deliberately adopt language that elides a distinction in a particular context, precisely because we don’t believe that distinction ought to be made in that context.
For example, in a context where I believe skin color ought not matter, I may use language that elides the distinction between skin colors. I may do this even if I care about that distinction: for example, if I observe that I do, in fact, care about my doctor’s skin color, but I don’t endorse caring about it, I might start using language that elides that distinction as a way of changing the degree to which I care about it.
So it seems worth asking whether, in the particular context you’re talking about, the connotations introduced by the term “preferences” are in fact unfortunate.
For instance, you class sexual preference among the “chocolate/vanilla” preferences for which the implication that they “shouldn’t really matter” is appropriate.
I would likely have agreed with you twenty years ago, when I had just broken up with my girlfriend and hadn’t yet started dating my current husband. OTOH, today I would likely “flee screaming” from a ray that made me heterosexual, since that would vastly decrease the value to me of my marriage.
Of course, you may object that this sort of practical consequence isn’t what you mean. But there are plenty of people who would “flee screaming” from a sexual-preference-altering ray for what they classify as moral reasons, without reference to practical consequences. And perhaps I’m one of them… after all, it’s not clear to me that my desire to preserve my marriage isn’t a “moral value.”
Indeed, it seems that there simply is no consistent fact of the matter as to whether my sexual preference is a “flee screaming” thing or not… it seems to depend on my situation. 20-year-old single me and 40-year-old married me disagree, and if tomorrow I were single again perhaps I’d once again change my mind.
Now, perhaps that just means that for me, sexual preference is a mere instrumental value, best understood in terms of what other benefits I get from it being one way or another, and is therefore a poor example of the distinction you’re getting at, and I should pick a different example.
On the other hand, just because I pick an different preference P such that I can’t imagine how a change in environment or payoff matrix might change P, doesn’t mean that P actually belongs in a different class from sexual preference. It might be equally true that a similarly pragmatic change would change P, I just can’t imagine the change that would do it.
Perhaps, under the right circumstances, I would not wish to flee from an autonomy/uniformity switching ray.
My point is that it’s not clear to me that it’s a mistake to elide over the distinction between moral values and aesthetic preferences. Maybe calling all of these things “preferences” is instead an excellent way of introducing the fortunate connotation that the degree to which any of them matter is equally arbitrary and situational, however intense the feeling that some preferences are “moral values” or “terminal values” or whatever other privileged term we want to apply to them.
20-year-old single me and 40-year-old married me disagree
These are two different people, many objections from the fact they disagree one ought to have from the fact that one and some random other contemporary person disagree.
And yet, a lot of our culture presumes that there are important differences between the two.
E.g., culturally we think it’s reasonable for someone at 20 to make commitments that are binding on that person at 40, whereas we think it’s really strange for someone at 20 or 40 to make commitments that are binding on some random other contemporary person.
Ah, sexual preference was a poor example in general– in my case, being single at the moment means I wouldn’t be injuring anybody if my preferences changed. Were I in a serious relationship, I’d flee from the ray gun too.
I personally don’t get that connotation from the term “preferences,” but I’m sure others do.
Anyway, so… Eliezer distinguishes prudential oughts from moral oughts by saying that moral oughts are what we ought to do to satisfy some small subset of our preferences: preferences that we wouldn’t want changed by an alien ray gun? I thought he was saying that I morally should_Luke do what will best satisfy a global consideration of my preferences.
No, no, no- I don’t mean that what I pointed out was the only distinction or the fundamental distinction, just that there’s a big honking difference in at least one salient way. I’m not speaking for Eliezer on what’s the best way to carve up that cluster in concept-space.
I certainly balk at the suggestion that there is a should_human, but I’d need to understand Eliezer in more detail on that point.
We’d need to do something specific with the world, there’s no reason any one person gets to have the privilege, and creating an agent for every human and having them fight it out is probably not the best possible solution.
I don’t think that adequately addresses lukeprog’s concern. Even granting that one person shouldn’t have the privilege of deciding the world’s fate, nor should an AI be created for every human to fight it out (although personally I don’t think an would-be FAI designer should rule these out as possible solutions just yet), that leaves many other possibilities for how to decide what to do with the world. I think the proper name for this problem is “should_AI_designer”, not “should_human”, and you need some other argument to justify the position that it makes sense to talk about “should_human”.
Between neurologically intact humans, there is indeed much cause to hope for overlap and coherence; and a great and reasonable doubt as to whether any present disagreement is really unresolvable, even it seems to be about “values”. The obvious reason for hope is the psychological unity of humankind, and the intuitions of symmetry, universalizability, and simplicity that we execute in the course of our moral arguments.
I’d appreciate clarification on what you mean by “You should_me maximize my preferences.”
I understand that the “objective” part is that we could both come to agree on the value of should_you and the value of should_me, but what do you mean when you say that I should_MattSimpson maximize your preferences?
I certainly balk at the suggestion that there is a should_human, but I’d need to understand Eliezer in more detail on that point.
And yes, if one’s own preferences are the foundation of ethics, most philosophers would simply call this subject matter practical rationality rather than morality. “Morality” is usually thought to be a term that refers to norms with a broader foundation and perhaps even “universal bindingness” or something. On this point, Eliezer just has an unusual way of carving up concept space that will confuse many people. (And this is coming from someone who rejects the standard analytic process of “conceptual analysis”, and is quite open to redefining terms to make them more useful and match the world more cleanly.)
Also, even if you think that the only reasons for action that exist come from relations between preferences and states of affairs, there are still ways to see morality as a system of hypothetical imperatives that is “broader” (and therefore may fit common use of the term “morality” better) than Eliezer’s meta-ethical theory. See for example Peter Railton or 1980s Philippa Foot or, well, Alonzo Fyfe and Luke Muehlhauser.
We already have a term that matches Eliezer’s use of “ought” and “should” quite nicely: it’s called the “prudential ought.” The term “moral ought” is usually applied to a different location in concept space, whether or not it successfully refers.
Anyway, are my remarks connecting with Eliezer’s actual stated position, do you think?
I mean that according to my preferences, you, me, and everyone else should maximize them. If you ask what should_MattSimpson be done, the short answer is maximize my preferences. Similarly, if you ask what should_lukeproq be done, the short answer is to maximize your preferences. It doesn’t matter who does the asking. If you ask should_agent should be done, you should maximize agent’s preferences. There is no “should” only should_agent’s. (Note, Eliezer calls should_human “should.” I think it’s an error of terminology, personally. It obscures his position somewhat).
Then Eliezer’s position is that all normativity is prudential normativity. But without the pop-culture connotations that come with this position. In other words, this doesn’t mean you can “do whatever you want.” You probably do, in fact, value other people, you’re a human after all. So murdering them is not ok, even if you know you can get away with it. (Note that this last conclusion might be salvageable even if there is no should_human.)
As for why Eliezer (and others here) think there is a should_human (or that human values are similar enough to talk about such a thing), the essence of the argument rests on ev-psych, but I don’t know the details beyond “ev-psych suggests that our minds would be very similar.”
Okay, that make sense.
Does Eliezer claim that murder is wrong for every agent? I find it highly likely that in certain cases, an agent’s murder of some person will best satisfy that agent’s preferences.
Murder is certainly not wrong_x for every agent x—we can think of an agent with a preference for people being murdered, even itself. However, it is almost always wrong_MattSimpson and (hopefully!) almost always wrong_lukeproq. So it depends on which question your are asking. If you’re asking “is murder wrong_human for every agent?” Eliezer would say yes. If you’re asking “is murder wrong_x for every agent x?” Eliezer would say no.
(I realize it was clear to both you and me which of the two you were asking, but for the benefit of confused readers, I made sure everything was clear)
I would be very surprised if EY gave those answers to those questions.
It seems pretty fundamental to his view of morality that asking about “wrong_human” and “wrong_x” is an important mis-step.
Maybe murder isn’t always wrong, but it certainly doesn’t depend (on EY’s view, as I understand it) on the existence of an agent with a preference for people being murdered (or the absence of such an agent).
That’s because for EY, “wrong” and “wrong_\human” mean the same thing. It’s semantics. When you ask “is X right or wrong?” in the every day sense of the term, you are actually asking “is X right_human or wrong_human?” But if murder is wrong_human, that doesn’t mean it’s wrong_clippy, for example. In both cases you are just checking a utility function, but different utility functions give different answers.
It seems clear from the metaethics posts is that if a powerful alien race comes along and converts humanity into paperclip-maximizers, such that making many paperclips comes to be right_human, EY would say that making many paperclips doesn’t therefore become right.
So it seems clear that at least under some circumstances, “wrong” and “wrong_human” don’t mean the same thing for EY, and that at least sometimes EY would say that “is X right or wrong?” doesn’t depend on what humans happen to want that day.
Now, if by “wrong_human” you don’t mean what humans would consider wrong the day you evaluate it, but rather what is considered wrong by humans today, then all of that is irrelevant to your claim.
In that case, yes, maybe you’re right that what you mean by “wrong_human” is also what EY means by “wrong.” But I still wouldn’t expect him to endorse the idea that what’s wrong or right depends in any way on what agents happen to prefer.
No one can change right_human, it’s a specific utility function. You can change the utility function that humans implement, but you can’t change right_human. That would be like changing e^x or 2 to something else. In other words, you’re right about what the metaethics posts say, and that’s what I’m saying too.
edit: or what jimrandomh said (I didn’t see his comment before I posted mine)
What if we use ‘human’ as a rigid designator for unmodified-human. Then in case aliens convert people into paperclip-maximizers, they’re no longer human, hence human_right no longer applies to them, but itself remains unchanged.
human_right still applies to them in the sense that they still should do what’s human_right. That’s the definition of should. (Remember, should refers to a specific set of terminal values, those that humans happen to have, called human_right) However, these modified humans, much like clippy, don’t care about human_right and so won’t be motivated to act based on human_right (except insofar as it helps make paperclips).
I’m not necessarily disagreeing with you because it’s a little ambiguous how you used the word “applies.” If you mean that the modified humans don’t care about human_right anymore, I agree. If you mean that the modified humans shouldn’t care about human_right, then I disagree.
I’m not sure why it’s necessary to use ‘should’ to mean morally_should, it could just be used to mean decision-theoretic_should. E.g. if you’re asked what a chess playing computer program should do to win a particular game, you could give a list of moves it should make. And when a human asks what they should do related to a moral question, you can first use the human_right function to determine what is the desired state of the world that they want to achieve, and then ask what you should do (as in decision-theoretic_should, or as in what moves/steps you need to execute, in analogy to the chess program) to create this state. Thus morality is contained within the human_right function and there’s no confusion over the meaning of ‘should’.
As long as you can keep the terms straight, sure. EY’s argument was that using “should” in that sense makes it easier to make mistakes related to relativism.
OK. At this point I must admit I’ve lost track of why these various suggestively named utility functions are of any genuine interest, so I should probably leave it there. Thanks for clarifying.
In that case, we would draw a distinction between right_unmodified_human and right_modified_human, and “right” would refer to the former.
Murder as I define it seems universally wrong_victim, but I doubt you could literally replace “victim” with any agent’s name.
I find the talk of “should_MattSimpson” very unpersuasive given the availability of alternative phrasings such as “approved_MattSimpson” or “valued_MattSimpson”. I have read below that EY discourages such talk, but it seems that’s for different reasons than mine. Could someone please point me to at least one post in the sequence which (almost/kinda/sorta) motivates such phrasings?
Alternate phrasings such as those you listed would probably be less confusing, i.e. replacing “should” in “should_X” with “valued” and reserving “should” for “valued_human”.
They would be missing some important distinctions between what we think of as our moral values and what we think of as “chocolate/vanilla” preferences. For one obvious example, consider an alien ray gun that ‘switches the way I feel’ about two things, X and Y, without otherwise affecting my utility function or anything else of value to me.
If X were, say, licorice jelly beans (yum) and Y were, say, buttered popcorn jelly beans (yuck), then I wouldn’t be too deeply bothered by the prospect of being zapped with this gun. (Same for sexual preference, etc.) But if X were “autonomy of individuals” and Y were “uniformity of individuals”, I would flee screaming from the prospect of being messed with that way, and would take some extreme actions (if I knew I’d be zapped) to prevent my new preferences from having large effects in the world.
Now we can develop whole theories about what this kind of difference consists in, but it’s at least relevant to the question of metaethics. In fact, I think that calling this wider class of volitions “preferences” is sneaking in an unfortunate connotation that they “shouldn’t really matter then”.
This sounds, to me, like it’s just the distinction between terminal and instrumental values. I don’t terminally value eating licorice jelly beans, I just like the way they taste and the feeling of pleasure they give me. If you switched the tastes of buttered popcorn jelly beans (yuck indeed) and licorice jelly beans, that would be fine by me. Hell, it would be an improvement since no one else likes that flavor (more for me!). The situation is NOT the same for “autonomy of individuals” and “uniformity of individuals” before I really do have terminal values for these things, apart from the way they make me feel.
How do you know that?
What would you expect to experience if your preference for individual autonomy in fact derived from something else?
It was meant as a hypothetical. I don’t actually know.
Ah. Sorry; I thought you were endorsing the idea.
Huh? You simply weigh “chocolate/vanilla” preferences differently than decisions that would affect goal-oriented agents.
I agree that by using a single term for the wider class of volitions—for example, by saying both that I “prefer” autonomy to uniformity and also that I “prefer” male sexual partners to female ones and also that I “prefer” chocolate to vanilla—I introduce the connotation that the distinctions between these various “preferences” aren’t important in the context of discourse.
To call that an unfortunate connotation is question-begging. Sometimes we deliberately adopt language that elides a distinction in a particular context, precisely because we don’t believe that distinction ought to be made in that context.
For example, in a context where I believe skin color ought not matter, I may use language that elides the distinction between skin colors. I may do this even if I care about that distinction: for example, if I observe that I do, in fact, care about my doctor’s skin color, but I don’t endorse caring about it, I might start using language that elides that distinction as a way of changing the degree to which I care about it.
So it seems worth asking whether, in the particular context you’re talking about, the connotations introduced by the term “preferences” are in fact unfortunate.
For instance, you class sexual preference among the “chocolate/vanilla” preferences for which the implication that they “shouldn’t really matter” is appropriate.
I would likely have agreed with you twenty years ago, when I had just broken up with my girlfriend and hadn’t yet started dating my current husband. OTOH, today I would likely “flee screaming” from a ray that made me heterosexual, since that would vastly decrease the value to me of my marriage.
Of course, you may object that this sort of practical consequence isn’t what you mean. But there are plenty of people who would “flee screaming” from a sexual-preference-altering ray for what they classify as moral reasons, without reference to practical consequences. And perhaps I’m one of them… after all, it’s not clear to me that my desire to preserve my marriage isn’t a “moral value.”
Indeed, it seems that there simply is no consistent fact of the matter as to whether my sexual preference is a “flee screaming” thing or not… it seems to depend on my situation. 20-year-old single me and 40-year-old married me disagree, and if tomorrow I were single again perhaps I’d once again change my mind.
Now, perhaps that just means that for me, sexual preference is a mere instrumental value, best understood in terms of what other benefits I get from it being one way or another, and is therefore a poor example of the distinction you’re getting at, and I should pick a different example.
On the other hand, just because I pick an different preference P such that I can’t imagine how a change in environment or payoff matrix might change P, doesn’t mean that P actually belongs in a different class from sexual preference. It might be equally true that a similarly pragmatic change would change P, I just can’t imagine the change that would do it.
Perhaps, under the right circumstances, I would not wish to flee from an autonomy/uniformity switching ray.
My point is that it’s not clear to me that it’s a mistake to elide over the distinction between moral values and aesthetic preferences. Maybe calling all of these things “preferences” is instead an excellent way of introducing the fortunate connotation that the degree to which any of them matter is equally arbitrary and situational, however intense the feeling that some preferences are “moral values” or “terminal values” or whatever other privileged term we want to apply to them.
These are two different people, many objections from the fact they disagree one ought to have from the fact that one and some random other contemporary person disagree.
And yet, a lot of our culture presumes that there are important differences between the two.
E.g., culturally we think it’s reasonable for someone at 20 to make commitments that are binding on that person at 40, whereas we think it’s really strange for someone at 20 or 40 to make commitments that are binding on some random other contemporary person.
Ah, sexual preference was a poor example in general– in my case, being single at the moment means I wouldn’t be injuring anybody if my preferences changed. Were I in a serious relationship, I’d flee from the ray gun too.
Thanks for this clarification.
I personally don’t get that connotation from the term “preferences,” but I’m sure others do.
Anyway, so… Eliezer distinguishes prudential oughts from moral oughts by saying that moral oughts are what we ought to do to satisfy some small subset of our preferences: preferences that we wouldn’t want changed by an alien ray gun? I thought he was saying that I morally should_Luke do what will best satisfy a global consideration of my preferences.
No, no, no- I don’t mean that what I pointed out was the only distinction or the fundamental distinction, just that there’s a big honking difference in at least one salient way. I’m not speaking for Eliezer on what’s the best way to carve up that cluster in concept-space.
Oh. Well, what do you think Eliezer has tried to say about how to carve up that cluster in concept-space?
We’d need to do something specific with the world, there’s no reason any one person gets to have the privilege, and creating an agent for every human and having them fight it out is probably not the best possible solution.
I don’t think that adequately addresses lukeprog’s concern. Even granting that one person shouldn’t have the privilege of deciding the world’s fate, nor should an AI be created for every human to fight it out (although personally I don’t think an would-be FAI designer should rule these out as possible solutions just yet), that leaves many other possibilities for how to decide what to do with the world. I think the proper name for this problem is “should_AI_designer”, not “should_human”, and you need some other argument to justify the position that it makes sense to talk about “should_human”.
I think Eliezer’s own argument is given here: