I don’t experience the emotions of moral outrage and moral approval whenever any of my preferences are hindered/satisfied—so it seems evident that my moral circuitry isn’t identical to my preference circuitry. It may overlap in parts, it may have fuzzy boundaries, but it’s not identical.
My own view is that morality is the brain’s attempt to extrapolate preferences about behaviours as they would be if you had no personal stakes/preferences about a situation.
So people don’t get morally outraged at other people eating chocolate icecreams, even when they personally don’t like chocolate icecreams, because they can understand that’s a strictly personal preference. If they believe it to be more than personal preference and make it into e.g. “divine commandment” or “natural law”, then moral outrage can occur.
That morality is a subjective attempt at objectivity explains many of the confusions people have about it.
The ice cream example is bad because the consequences are purely internal to the person consuming the ice cream. What if the chocolate ice cream was made with slave labour? Many people would then object to you buying it on moral grounds.
Eliezer has produced an argument I find convincing that morality is the back propagation of preference to the options of an intermediate choice. That is to say, it is “bad” to eat chocolate ice cream because it economically supports slavers, and I prefer a world without slavery. But if I didn’t know about the slave-labour ice cream factory, my preference would be that all-things-being-equal you get to make your own choices about what you eat, and therefore I prefer that you choose (and receive) the one you want, which is your determination to make, not mine.
Do you agree with EY’s essay on the nature of right-ness which I linked to?
Do you think FAI will need to treat morality differently from other preferences?
I would prefer a AI that followed my extrapolated preferences, than a AI that followed my morality. But a AI that followed my morality would be morally superior to an AI that followed my extrapolated preferences.
If you don’t understand the distinction I’m making above, consider a case of the AI having to decide whether to save my own child vs saving a thousand random other children. I’d prefer the former, but I believe the latter would be the morally superior choice.
Is that idea really so hard to understand? Would you dismiss the distinction I’m making as merely colloquial language?
If you don’t understand the distinction I’m making above, consider a case of the AI having to decide whether to save my own child vs saving a thousand random other children. I’d prefer the former, but I believe the latter would be the morally superior choice.
Wow there is so much wrapped up in this little consideration. The heart of the issue is that we (by which I mean you, but I share your delimma) have truly conflicting preferences.
Honestly I think you should not be afraid to say that saving your own child is the moral thing to do. And you don’t have to give excuses either—it’s not that “if everyone saved their own child, then everyone’s child will be looked after” or anything like that. No, the desire to save your own child is firmly rooted in our basic drives and preferences, enough so that we can go quite far in calling it a basic foundational moral axiom. It’s not actually axiomatic, but we can safely treat it as such.
At the same time we have a basic preference to seek social acceptance and find commonality with the people we let into our lives. This drives us to want outcomes that are universally or at least most-widely acceptable, and seek moral frameworks like utilitarianism which lead to these outcomes. Usually this drive is secondary to self-serving preferences for most people, and that is OK.
For some reason you’ve called making decisions in favor of self-serving drives “preferences” and decisions in favor of social drives “morality.” But the underlying mechanism is the same.
“But wait, if I choose self-serving drives over social conformity, doesn’t that lead to me to make the decision to save one life in exclusion to 1000 others?” Yes, yes it does. This massive sub-thread started with me objecting to the idea that some “friendly” AI somewhere could derive morality experimentally from my preferences or the collective preferences of humankind, make it consistent, apply the result universally, and that I’d be OK with that outcome. But that cannot work because there is not, and cannot be a universal morality that satisfies everyone—every one of those thousand other children have parents that want their kid to survive and would see your child dead if need be.
Honestly I think you should not be afraid to say that saving your own child is the moral thing to do
What do you mean by “should not”?
and that is OK.
What do you mean by “OK”?
For some reason you’ve called making decisions in favor of self-serving drives “preferences” and decisions in favor of social drives “morality.” But the underlying mechanism is the same.
Show me the neurological studies that prove it.
But that cannot work because there is not, and cannot be a universal morality that satisfies everyone—every one of those thousand other children have parents that want their kid to survive and would see your child dead if need be.
Yes, and yet if none of the children were mine, and if I wasn’t involved in the situation at all, I would say “save the 1000 children rather than the 1”. And if someone else, also not personally involved, could make the choice and chose to flip a coin instead in order to decide, I’d be morally outraged at them.
You can now give me a bunch of reasons of why this is just preference, while at the same time EVERYTHING about it (how I arrive to my judgment, how I feel about the judgment of others) makes it a whole distinct category of its own. I’m fine with abolishing useless categories when there’s no meaningful distinction, but all you people should stop trying to abolish categories where there pretty damn obviously IS one.
I suspect that he means something like ’Even though utilitarianism (on LW) and altruism (in general) are considered to be what morality is, you should not let that discourage you from asserting that selfishly saving your own child is the right thing to do”. (Feel free to correct me if I’m wrong.)
I’m fine with abolishing useless categories when there’s no meaningful distinction, but all you people should stop trying to abolish categories where there pretty damn obviously IS one.
I’ve explained to you twice now how the two underlying mechanisms are unified, and pointed to Eliezer’s quite good explanation on the matter. I don’t see the need to go through that again.
I would prefer a AI that followed my extrapolated preferences, than a AI that followed my morality. But a AI that followed my morality would be morally superior to an AI that followed my extrapolated preferences.
If you were offered a bunch of AIs with equivalent power, but following different mixtures of your moral and non-moral preferences, which one would you run? (I guess you’re aware of the standard results saying a non-stupid AI must follow some one-dimensional utility function, etc.)
If you were offered a bunch of AIs with equivalent power, but following different mixtures of your moral and non-moral preferences, which one would you run?
I guess whatever ratio of my moral and non-moral preferences best represents their effect on my volition.
My related but different thoughts here. In particular, I don’t agree that emotions like moral outrage and approval are impersonal, though I agree that we often justify those emotions using impersonal language and beliefs.
I didn’t say that moral outrage and approval are impersonal. Obviously nothing that a person does can truly be “impersonal”. But it may be an attempt at impersonality.
The attempt itself provides a direction that significantly differentiates between moral preferences and non-moral preferences.
I didn’t mean some idealized humanly-unrealizable notion of impersonality, I meant the thing we ordinarily use “impersonal” to mean when talking about what humans do.
I don’t experience the emotions of moral outrage and moral approval whenever any of my preferences are hindered/satisfied—so it seems evident that my moral circuitry isn’t identical to my preference circuitry. It may overlap in parts, it may have fuzzy boundaries, but it’s not identical.
My own view is that morality is the brain’s attempt to extrapolate preferences about behaviours as they would be if you had no personal stakes/preferences about a situation.
So people don’t get morally outraged at other people eating chocolate icecreams, even when they personally don’t like chocolate icecreams, because they can understand that’s a strictly personal preference. If they believe it to be more than personal preference and make it into e.g. “divine commandment” or “natural law”, then moral outrage can occur.
That morality is a subjective attempt at objectivity explains many of the confusions people have about it.
The ice cream example is bad because the consequences are purely internal to the person consuming the ice cream. What if the chocolate ice cream was made with slave labour? Many people would then object to you buying it on moral grounds.
Eliezer has produced an argument I find convincing that morality is the back propagation of preference to the options of an intermediate choice. That is to say, it is “bad” to eat chocolate ice cream because it economically supports slavers, and I prefer a world without slavery. But if I didn’t know about the slave-labour ice cream factory, my preference would be that all-things-being-equal you get to make your own choices about what you eat, and therefore I prefer that you choose (and receive) the one you want, which is your determination to make, not mine.
Do you agree with EY’s essay on the nature of right-ness which I linked to?
That doesn’t seem to be required for Eliezer’s argument...
I guess the relevant question is, do you think FAI will need to treat morality differently from other preferences?
I would prefer a AI that followed my extrapolated preferences, than a AI that followed my morality. But a AI that followed my morality would be morally superior to an AI that followed my extrapolated preferences.
If you don’t understand the distinction I’m making above, consider a case of the AI having to decide whether to save my own child vs saving a thousand random other children. I’d prefer the former, but I believe the latter would be the morally superior choice.
Is that idea really so hard to understand? Would you dismiss the distinction I’m making as merely colloquial language?
Wow there is so much wrapped up in this little consideration. The heart of the issue is that we (by which I mean you, but I share your delimma) have truly conflicting preferences.
Honestly I think you should not be afraid to say that saving your own child is the moral thing to do. And you don’t have to give excuses either—it’s not that “if everyone saved their own child, then everyone’s child will be looked after” or anything like that. No, the desire to save your own child is firmly rooted in our basic drives and preferences, enough so that we can go quite far in calling it a basic foundational moral axiom. It’s not actually axiomatic, but we can safely treat it as such.
At the same time we have a basic preference to seek social acceptance and find commonality with the people we let into our lives. This drives us to want outcomes that are universally or at least most-widely acceptable, and seek moral frameworks like utilitarianism which lead to these outcomes. Usually this drive is secondary to self-serving preferences for most people, and that is OK.
For some reason you’ve called making decisions in favor of self-serving drives “preferences” and decisions in favor of social drives “morality.” But the underlying mechanism is the same.
“But wait, if I choose self-serving drives over social conformity, doesn’t that lead to me to make the decision to save one life in exclusion to 1000 others?” Yes, yes it does. This massive sub-thread started with me objecting to the idea that some “friendly” AI somewhere could derive morality experimentally from my preferences or the collective preferences of humankind, make it consistent, apply the result universally, and that I’d be OK with that outcome. But that cannot work because there is not, and cannot be a universal morality that satisfies everyone—every one of those thousand other children have parents that want their kid to survive and would see your child dead if need be.
What do you mean by “should not”?
What do you mean by “OK”?
Show me the neurological studies that prove it.
Yes, and yet if none of the children were mine, and if I wasn’t involved in the situation at all, I would say “save the 1000 children rather than the 1”. And if someone else, also not personally involved, could make the choice and chose to flip a coin instead in order to decide, I’d be morally outraged at them.
You can now give me a bunch of reasons of why this is just preference, while at the same time EVERYTHING about it (how I arrive to my judgment, how I feel about the judgment of others) makes it a whole distinct category of its own. I’m fine with abolishing useless categories when there’s no meaningful distinction, but all you people should stop trying to abolish categories where there pretty damn obviously IS one.
I suspect that he means something like ’Even though utilitarianism (on LW) and altruism (in general) are considered to be what morality is, you should not let that discourage you from asserting that selfishly saving your own child is the right thing to do”. (Feel free to correct me if I’m wrong.)
Yes that is correct.
So you explained “should not” by using a sentence that also has “should not” in it.
I hope it’s a more clear “should not”.
I’ve explained to you twice now how the two underlying mechanisms are unified, and pointed to Eliezer’s quite good explanation on the matter. I don’t see the need to go through that again.
If you were offered a bunch of AIs with equivalent power, but following different mixtures of your moral and non-moral preferences, which one would you run? (I guess you’re aware of the standard results saying a non-stupid AI must follow some one-dimensional utility function, etc.)
I guess whatever ratio of my moral and non-moral preferences best represents their effect on my volition.
My related but different thoughts here. In particular, I don’t agree that emotions like moral outrage and approval are impersonal, though I agree that we often justify those emotions using impersonal language and beliefs.
I didn’t say that moral outrage and approval are impersonal. Obviously nothing that a person does can truly be “impersonal”. But it may be an attempt at impersonality.
The attempt itself provides a direction that significantly differentiates between moral preferences and non-moral preferences.
I didn’t mean some idealized humanly-unrealizable notion of impersonality, I meant the thing we ordinarily use “impersonal” to mean when talking about what humans do.