It seems to me that EY himself addressed all three of the objections you list (though of course this doesn’t imply he addressed them adequately).
(1) It’s incompatible with the datum that substantive, fundamental normative disagreement is in fact possible. People may share the concept of a normative reason, even if they fundamentally disagree about which features of actions are the ones that give us reasons.
My own thinking is that humans tend to have the same underlying (evolved) structures behind our hard-to-articulate meta-ethical heuristics, even when we disagree broadly on object-level ethical issues (and of course hand-pick our articulations of the meta-criteria to support our object-level beliefs- the whole machinery of bias applies here).
This implies both that my object-level beliefs can be at odds with their meta-level criteria (if this becomes too obvious for me to rationalize away, I’m more likely to change one or other object-level belief than to change the meta-level heuristic), and that you and I can disagree fundamentally on the object level while still believing that there’s something in common which makes argumentation relevant to our disagreement.
OK. I’ll reply here because if I reply there, you won’t get the notifications.
The crux of your argument, it seems to me, is the following intuition:
Rather, it is essential to the concept of morality that it involves shared standards common to all fully reasonable agents.
This is certainly a property we would want morality to have, and one which human beings naturally assume it must have– but is that the central property of it? Should it turn out that nothing which looks like morality has this property, does it logically follow that all morality is dead, or is that reaction just a human impulse?
(I will note, with all the usual caveats, that believing one’s moral sentiments to be universal in scope and not based on preference is a big advantage in object-level moral arguments, and that we happen to be descended from the winners of arguments about tribal politics and morality.)
If a certain set of moral impulses involves shared standards common to, say, every sane human being, then moral arguments would still work among those human beings, in exactly the way you would want them to work across all intelligent beings. Frankly, that’s good enough for me. Why give baby-eating aliens in another universe veto powers over every moral intuition of yours?
Thanks for the reply—I find this a very interesting topic. One thing I should clarify is that my view doesn’t entail giving aliens “veto powers”, as you put it; an alternative response is to take them to be unreasonable to intrinsically desire the eating of babies. That isn’t an intrinsically desirable outcome (I take it), i.e. there is no reason to desire such a thing. Stronger still, we may think it intrinsically undesirable, so that insofar as an agent has such desires they are contrary to reason. (This requires a substantive notion of reason that goes beyond mere instrumental rationality, of course.)
In any case, I’d put the crux of my argument slightly differently. The core intuition is just that it’s possible to have irresolvable moral disagreements. We can imagine a case where Bob is stubbornly opposed to abortion, and Jane is just as stubbornly in favour of it, and neither agent is disposed to change their mind in light of any additional information. EY’s view would seem to imply that the two agents mustn’t really disagree. And that just seems a mistake: it’s part of our concept of morality that this very conceptcould be shared by someone who fundamentally (and irresolvably) disagrees with us about what the substantive moral facts are. This is because we’re aspiring to conform our judgments to a standard that is outside of ourselves. (If you don’t think there are any such objective standards, then that’s just to say that there are no normative facts, given my concept of normativity.)
Human beings are analogous to computers. Morality and other aspects of behavior and cognition are analogous to programs. It is a type error to ask whether a program “really exists” somewhere outside a computer, or is “intrinsic” to a computer, or is “contingent”, or something like that. Such questions don’t correspond to observations within the world that could turn out one way or the other. You see a computer running a certain program and that’s the end of the story.
Your mind is a program too, and your moral intuitions are how your algorithm feels from inside, not a direct perception of external reality (human beings are physically incapable of that kind of thing, though they may feel otherwise). I know for a fact that you have no astral gate in your head to pull answers from the mysterious source of morality. But this doesn’t imply that your moral intuitions “should” be worthless to you and you “should” seek external authority! There’s nothing wrong with mankind living by its internal moral lights.
Yes, it’s possible that different computers will have different programs. Our world contains billions of similar “moist robots” running similar programs, perhaps because we were all created from design documents that are 99% identical for historical reasons, and also because we influence each other a lot. Your intuition that all “possible” sentient agents must share a common morality is unlikely to survive an encounter with any sentient agent that’s substantially different from a human. We can imagine such agents easily, e.g. a machine that will search for proofs to Goldbach’s conjecture and turn surrounding matter and energy into computing machinery to that end. Such a machine may be more ingenious than any human in creating other machines, discovering new physics, etc., but will never gravitate toward your intuition that one shouldn’t kill babies. Most possible “intelligent agents” (aka algorithms that can hit small targets in large search spaces) aren’t humans in funny suits.
I expect Richard’s memeset has understanding of all your points that doesn’t move his current position. You’re probably exposing him to arguments he has already encountered, so there’s little point in expecting a different result. I’m not saying that Richard can’t be moved by argument, just not by standard argument that is already known to have failed to move him. He even probably “agrees” with a lot of your points, just with a different and more sophisticated understanding than yours.
On the other hand, it might work for the benefit of more naive onlookers.
The intent of my comment wasn’t to convince Richard (I never do that), but to sharpen our points and make him clarify whatever genuine insight he possesses and we don’t.
Your last paragraph suggests that you’ve misunderstood my view. I’m not making an empirical claim to the effect that all agents will eventually converge to our values—I agree that that’s obviously false. I don’t even think that all formally intelligent agents are guaranteed to have normative concepts like ‘ought’, ‘reason’, or ‘morality’. The claim is just that such a radically different agent could share our normative concepts (in particular, our aspiration to a mind-independent standard), even if they would radically disagree with us about which things fall under the concept. We could both have full empirical knowledge about our own and each other’s desires/dispositions, and yet one (or both) of us might be wrong about what we really have reason to want and to do.
(Aside: the further claim about “reasons” in your last sentence presupposes a subjectivist view about reasons that I reject.)
What use is this concept of “reasonability”? Let’s say I build an agent that wants to write the first 1000 Fibonacci numbers in mile-high digits on the Moon, except skipping the 137th one. When you start explaining to the agent that it’s an “arbitrary omission” and it “should” amend its desires for greater “consistency”, the agent just waves you off because listening to you isn’t likely to further its current goals. Listening to you is not rational for the agent in the sense that most people on LW use the term: it doesn’t increase expected utility. If by “rational” you mean something else, I’d like to understand what exactly.
I mean ‘rational’ in the ordinary, indefinable sense, whereby calling a decision ‘irrational’ expresses a distinctive kind of criticism—similar to that expressed by the words ‘crazy’, ‘foolish’, ‘unwise’, etc. (By contrast, you can just say “maximizes expected utility” if you really mean nothing more than maximizes expected utility—but note that that’s a merely descriptive concept, not a normative one.)
If you don’t possess this concept—if you never have thoughts about what’s rational, over and above just what maximizes expected utility—then I can’t help you.
When we’re trying to reduce intuitions, there’s no helping starting from informal ideas. Another question is that it’s not proper to stop there, but Richard doesn’t exactly suggest that.
A more salient to me question is, What are suggestions about redefining this “rational” intuitive idea good for, if it’s supposed to be the source material? Such question even explains how the idea of considering, say, “consciousness”, in a more precise sense is methodologically a step in the wrong direction: when words are the data you work with, you should be careful to assign new words to new ideas used for analyzing them.
I’m not sure I understand your second paragraph. Are you suggesting that if we come up with a new theory to explain some aspect of consciousness, we should use a word other than “consciousness” in that theory, to avoid potentially losing some of our intuitions about consciousness?
First, EY makes it abundantlyclear that two agents can have a fundamental disagreement on values– it’s just not the best (or most helpful) assumption when you’re talking about two sane human beings with a vast sea of common frameworks and heuristics.
Secondly, I’m worried about what you’re trying to do with words when you suggest we “take them to be unreasonable to intrinsically desire the eating of babies”.
If you’re making an empirical claim that an alien with fundamentally different terminal values will (say) be uninterested in negotiating mutually beneficial deals, or will make patently suboptimal decisions by its own criteria, or exhibit some other characteristic of what we mean by “unreasonable”, then you’d need some strong evidence for that claim.
If instead you openly redefine “reasonable” to include “shares our fundamental moral standards”, then the property
it is essential to the concept of morality that it involves shared standards common to all fully reasonable agents
becomes a tautology which no longer excludes “meta-semantic subjectivism”, as you put it. So I’m puzzled what you mean.
Talking past each other a bit here. Let me try again.
EY makes it abundantly clear that two agents can have a fundamental disagreement on values
EY allows for disagreement in attitude: you might want one thing, while the babyeaters want something different. Of course I’m not charging him with being unable to accommodate this. The objection is instead that he’s unable to accommodate disagreement in moral judgment (at the fundamental level). Normativity as mere semantics, and all that.
Your second point rests on a false dichotomy. I’m not making an empirical claim, but nor am I merely defining the word “reasonable”. Rather, I’m making a substantive normative (non-empirical) hypothesis about which things are reasonable. If you can’t make sense of the idea of a substantive non-empirical issue, you may have fallen victim to scientism.
The core intuition is just that it’s possible to have irresolvable moral disagreements.
What is the difference between an in-principle irresolvable disagreement (moral or otherwise), and talking past each other (i.e. talking of different subject matters, or from different argument-processing frameworks)?
an alternative response is to take them to be unreasonable to intrinsically desire the eating of babies
What fact have you established by manipulating the definition of a word in this manner? I want a meta-ethical theory that at least describes baby-eaters, because I don’t expect to have object-level understanding of human morality that is substantially more accurate than what you’d get if you add baby-eating impulses to it.
It seems to me that EY himself addressed all three of the objections you list (though of course this doesn’t imply he addressed them adequately).
Moral Error and Moral Disagreement confronts this.
My own thinking is that humans tend to have the same underlying (evolved) structures behind our hard-to-articulate meta-ethical heuristics, even when we disagree broadly on object-level ethical issues (and of course hand-pick our articulations of the meta-criteria to support our object-level beliefs- the whole machinery of bias applies here).
This implies both that my object-level beliefs can be at odds with their meta-level criteria (if this becomes too obvious for me to rationalize away, I’m more likely to change one or other object-level belief than to change the meta-level heuristic), and that you and I can disagree fundamentally on the object level while still believing that there’s something in common which makes argumentation relevant to our disagreement.
Yeah, I’m the “Richard4” in the comments thread there :-)
OK. I’ll reply here because if I reply there, you won’t get the notifications.
The crux of your argument, it seems to me, is the following intuition:
This is certainly a property we would want morality to have, and one which human beings naturally assume it must have– but is that the central property of it? Should it turn out that nothing which looks like morality has this property, does it logically follow that all morality is dead, or is that reaction just a human impulse?
(I will note, with all the usual caveats, that believing one’s moral sentiments to be universal in scope and not based on preference is a big advantage in object-level moral arguments, and that we happen to be descended from the winners of arguments about tribal politics and morality.)
If a certain set of moral impulses involves shared standards common to, say, every sane human being, then moral arguments would still work among those human beings, in exactly the way you would want them to work across all intelligent beings. Frankly, that’s good enough for me. Why give baby-eating aliens in another universe veto powers over every moral intuition of yours?
Thanks for the reply—I find this a very interesting topic. One thing I should clarify is that my view doesn’t entail giving aliens “veto powers”, as you put it; an alternative response is to take them to be unreasonable to intrinsically desire the eating of babies. That isn’t an intrinsically desirable outcome (I take it), i.e. there is no reason to desire such a thing. Stronger still, we may think it intrinsically undesirable, so that insofar as an agent has such desires they are contrary to reason. (This requires a substantive notion of reason that goes beyond mere instrumental rationality, of course.)
In any case, I’d put the crux of my argument slightly differently. The core intuition is just that it’s possible to have irresolvable moral disagreements. We can imagine a case where Bob is stubbornly opposed to abortion, and Jane is just as stubbornly in favour of it, and neither agent is disposed to change their mind in light of any additional information. EY’s view would seem to imply that the two agents mustn’t really disagree. And that just seems a mistake: it’s part of our concept of morality that this very concept could be shared by someone who fundamentally (and irresolvably) disagrees with us about what the substantive moral facts are. This is because we’re aspiring to conform our judgments to a standard that is outside of ourselves. (If you don’t think there are any such objective standards, then that’s just to say that there are no normative facts, given my concept of normativity.)
Richard, hello.
Human beings are analogous to computers. Morality and other aspects of behavior and cognition are analogous to programs. It is a type error to ask whether a program “really exists” somewhere outside a computer, or is “intrinsic” to a computer, or is “contingent”, or something like that. Such questions don’t correspond to observations within the world that could turn out one way or the other. You see a computer running a certain program and that’s the end of the story.
Your mind is a program too, and your moral intuitions are how your algorithm feels from inside, not a direct perception of external reality (human beings are physically incapable of that kind of thing, though they may feel otherwise). I know for a fact that you have no astral gate in your head to pull answers from the mysterious source of morality. But this doesn’t imply that your moral intuitions “should” be worthless to you and you “should” seek external authority! There’s nothing wrong with mankind living by its internal moral lights.
Yes, it’s possible that different computers will have different programs. Our world contains billions of similar “moist robots” running similar programs, perhaps because we were all created from design documents that are 99% identical for historical reasons, and also because we influence each other a lot. Your intuition that all “possible” sentient agents must share a common morality is unlikely to survive an encounter with any sentient agent that’s substantially different from a human. We can imagine such agents easily, e.g. a machine that will search for proofs to Goldbach’s conjecture and turn surrounding matter and energy into computing machinery to that end. Such a machine may be more ingenious than any human in creating other machines, discovering new physics, etc., but will never gravitate toward your intuition that one shouldn’t kill babies. Most possible “intelligent agents” (aka algorithms that can hit small targets in large search spaces) aren’t humans in funny suits.
I expect Richard’s memeset has understanding of all your points that doesn’t move his current position. You’re probably exposing him to arguments he has already encountered, so there’s little point in expecting a different result. I’m not saying that Richard can’t be moved by argument, just not by standard argument that is already known to have failed to move him. He even probably “agrees” with a lot of your points, just with a different and more sophisticated understanding than yours.
On the other hand, it might work for the benefit of more naive onlookers.
The intent of my comment wasn’t to convince Richard (I never do that), but to sharpen our points and make him clarify whatever genuine insight he possesses and we don’t.
That’s a motivation I didn’t consider. (Agreed.)
Yeah, as Vladimir guessed, this is all familiar.
Your last paragraph suggests that you’ve misunderstood my view. I’m not making an empirical claim to the effect that all agents will eventually converge to our values—I agree that that’s obviously false. I don’t even think that all formally intelligent agents are guaranteed to have normative concepts like ‘ought’, ‘reason’, or ‘morality’. The claim is just that such a radically different agent could share our normative concepts (in particular, our aspiration to a mind-independent standard), even if they would radically disagree with us about which things fall under the concept. We could both have full empirical knowledge about our own and each other’s desires/dispositions, and yet one (or both) of us might be wrong about what we really have reason to want and to do.
(Aside: the further claim about “reasons” in your last sentence presupposes a subjectivist view about reasons that I reject.)
What use is this concept of “reasonability”? Let’s say I build an agent that wants to write the first 1000 Fibonacci numbers in mile-high digits on the Moon, except skipping the 137th one. When you start explaining to the agent that it’s an “arbitrary omission” and it “should” amend its desires for greater “consistency”, the agent just waves you off because listening to you isn’t likely to further its current goals. Listening to you is not rational for the agent in the sense that most people on LW use the term: it doesn’t increase expected utility. If by “rational” you mean something else, I’d like to understand what exactly.
I mean ‘rational’ in the ordinary, indefinable sense, whereby calling a decision ‘irrational’ expresses a distinctive kind of criticism—similar to that expressed by the words ‘crazy’, ‘foolish’, ‘unwise’, etc. (By contrast, you can just say “maximizes expected utility” if you really mean nothing more than maximizes expected utility—but note that that’s a merely descriptive concept, not a normative one.)
If you don’t possess this concept—if you never have thoughts about what’s rational, over and above just what maximizes expected utility—then I can’t help you.
I don’t think we can make progress with such imprecise thinking. Eliezer has a nice post about that.
When we’re trying to reduce intuitions, there’s no helping starting from informal ideas. Another question is that it’s not proper to stop there, but Richard doesn’t exactly suggest that.
A more salient to me question is, What are suggestions about redefining this “rational” intuitive idea good for, if it’s supposed to be the source material? Such question even explains how the idea of considering, say, “consciousness”, in a more precise sense is methodologically a step in the wrong direction: when words are the data you work with, you should be careful to assign new words to new ideas used for analyzing them.
In this old comment, he does seem to suggest stopping there.
I’m not sure I understand your second paragraph. Are you suggesting that if we come up with a new theory to explain some aspect of consciousness, we should use a word other than “consciousness” in that theory, to avoid potentially losing some of our intuitions about consciousness?
Yes.
First, EY makes it abundantly clear that two agents can have a fundamental disagreement on values– it’s just not the best (or most helpful) assumption when you’re talking about two sane human beings with a vast sea of common frameworks and heuristics.
Secondly, I’m worried about what you’re trying to do with words when you suggest we “take them to be unreasonable to intrinsically desire the eating of babies”.
If you’re making an empirical claim that an alien with fundamentally different terminal values will (say) be uninterested in negotiating mutually beneficial deals, or will make patently suboptimal decisions by its own criteria, or exhibit some other characteristic of what we mean by “unreasonable”, then you’d need some strong evidence for that claim.
If instead you openly redefine “reasonable” to include “shares our fundamental moral standards”, then the property
becomes a tautology which no longer excludes “meta-semantic subjectivism”, as you put it. So I’m puzzled what you mean.
Talking past each other a bit here. Let me try again.
EY allows for disagreement in attitude: you might want one thing, while the babyeaters want something different. Of course I’m not charging him with being unable to accommodate this. The objection is instead that he’s unable to accommodate disagreement in moral judgment (at the fundamental level). Normativity as mere semantics, and all that.
Your second point rests on a false dichotomy. I’m not making an empirical claim, but nor am I merely defining the word “reasonable”. Rather, I’m making a substantive normative (non-empirical) hypothesis about which things are reasonable. If you can’t make sense of the idea of a substantive non-empirical issue, you may have fallen victim to scientism.
What is the difference between an in-principle irresolvable disagreement (moral or otherwise), and talking past each other (i.e. talking of different subject matters, or from different argument-processing frameworks)?
What fact have you established by manipulating the definition of a word in this manner? I want a meta-ethical theory that at least describes baby-eaters, because I don’t expect to have object-level understanding of human morality that is substantially more accurate than what you’d get if you add baby-eating impulses to it.
Ah! Sorry for carrying coals to Newcastle, then. Let me catch up in that thread.