Eliezer’s metaethics might be clarified in terms of the distinctions between sense, reference, and reference-fixing descriptions. I take it that Eliezer wants to use ‘right’ as a rigid designator to denote some particular set of terminal values, but this reference fact is fixed by means of a seemingly ‘relative’ procedure (namely, whatever terminal values the speaker happens to hold, on some appropriate [if somewhat mysterious] idealization). Confusions arise when people mistakenly read this metasemantic subjectivism into the first-order semantics or meaning of ‘right’.
In summary:
(i) ‘Right’ means, roughly, ‘promotes external goods X, Y and Z’
(ii) claim i above is true because I desire X, Y, and Z.
Note that Speakers Use Their Actual Language, so murder would still be wrong even if I had the desires of a serial killer. But if I had those violent terminal values, I would speak a slightly different language than I do right now, so that when KillerRichard asserts “Murder is right!” what he says is true. We don’t really disagree, but are instead merely talking past each other.
Virtues of the theory:
(a) By rigidifying on our actual, current desires (or idealizations thereupon), it avoids Inducing Desire Satisfactions.
(b) Shifting the subjectivity out to the metasemantic level leaves us with a first-order semantic proposal that at least does a better job than simple subjectivism at ‘saving the phenomena’. (It has echoes of Mark Schroeder’s desire-based view of reasons, according to which the facts that give us reasons are the propositional contents of our desires, rather than the desires themselves. Or something like that.)
(c) It’s naturalistic, if you find moral non-naturalism ‘spooky’. (Though I’d sooner recommend Mackie-style error theory for naturalists, since I don’t think (b) above is enough to save the phenomena.)
Objections
(1) It’s incompatible with the datum that substantive, fundamental normative disagreement is in fact possible. People may share the concept of a normative reason, even if they fundamentally disagree about which features of actions are the ones that give us reasons.
(2) The semantic tricks merely shift the lump under the rug, they don’t get rid of it. Standard worries about relativism re-emerge, e.g. an agent can know a priori that their own fundamental values are right, given how the meaning of the word ‘right’ is determined. This kind of (even merely ‘fundamental’) infallibility seems implausible.
(3) Just as simple subjectivism is an implausible theory of what ‘right’ means, so Eliezer’s meta-semantic subjectivism is an implausible theory of why ‘right’ means promoting external goods X, Y, Z. An adequately objective metaethics shouldn’t even give preferences a reference-fixing role.
I think this is an excellent summary. I would make the following comments:
Confusions arise when people mistakenly read this metasemantic subjectivism into the first-order semantics or meaning of ‘right’.
Yes, but I think Eliezer was mistaken in identifying this kind of confusion as the fundamental source of the objections to his theory (as in the Löb’s theorem discussion). Sophisticated readers of LW (or OB, at the time) are surely capable of distinguishing between logical levels. At least, I am—but nevertheless, I still didn’t feel that his theory was adequately “non-relativist” to satisfy the kinds of people who worry about “relativism”. What I had in mind, in other words, was your objections (2) and (3).
The answer to those objections, by the way, is that an “adequately objective” metaethics is impossible: the minds of complex agents (such as humans) are the only place in the universe where information about morality is to be found, and there are plenty of possible minds in mind-design space (paperclippers, pebblesorters, etc.) from which it is impossible to extract the same information. This directly answers (3), anyway; as for (2), “fallibility” is rescued (on the object level) by means of imperfect introspective knowledge: an agent could be mistaken about what its own terminal values are.
Note that your answer to (2) also answers (1): value uncertainty makes it seem as if there is substantive, fundamental normative disagreement even if there isn’t. (Or maybe there is if you don’t buy that particular element of EY’s theory)
The answer to those objections, by the way, is that an “adequately objective” metaethics is impossible: the minds of complex agents (such as humans) are the only place in the universe where information about morality is to be found, and there are plenty of possible minds in mind-design space (paperclippers, pebblesorters, etc.) from which it is impossible to extract the same information.
Elizer attempted to deal with that problem by defining a certain set of things as “h-right”, that is, morally right from the frame of reference of the human mind. He made clear that alien entities probably would not care about what is h-right, but that humans do, and that’s good enough.
The answer to those objections, by the way, is that an “adequately objective” metaethics is impossible
That’s not a reason to prefer EY’s theory to an error theory (according to which properly normative properties would have to be irreducibly normative, but no such properties actually exist).
Until persuaded otherwise, I agree with you on this point. (These days, I take Richard Joyce to have the clearest defense of error theory, and I just subtract his confusing-to-me defense of fictionalism.) Besides, I think there are better ways of getting something like an ‘objective’ ethical theory (in something like a ‘realist’ sense) while still holding that reasons for action arise only from desires, or from relations between desires and states of affairs. In fact, that’s the kind of theory I defend: desirism. Though, I’m not too interested anymore in whether desirism is to be called ‘objective’ or ‘realist’, even though I think a good case can be made for both.
You’re speaking my language, thanks! I hope this is EY’s view, because I know what this means. Maybe now I can go back and read EY’s sequence in light of this interpretation and it will make more sense to me now.
EY’s theory as presented above makes me suspicious that making basic evaluative moral terms rigid designators is a kind of ‘trick’ which, though perhaps not intended, very easily has the effect of carrying along some common absolutist connotations of those terms where they no longer apply in EY’s use of those terms.
At the moment, I’m not so worried about objection (1), but objections (2) and (3) are close to what bother me about EY’s theory, especially if this is foundational for EY’s thinking about how we ought to be designing a Friendly AI. If we’re working on a project as important as Friendly AI, it becomes an urgent problem to get our meta-ethics right, and I’m not sure Eliezer has done it yet. Which is why we need more minds working on this problem. I hope to be one of those minds, even if my current meta-ethics turns out to be wrong (I’ve held my current meta-ethics for under 2 years, anyway, and it has shifted slightly since adoption).
But, at the moment it remains plausible to me that Eliezer is right, and I just don’t see why right now. Eliezer is a very smart guy who has invested a lot of energy into training himself to think straight about things and respond to criticism either with adequate counterargument or by dropping the criticized belief.
invested a lot of energy into training himself to think straight about things and respond to criticism either with adequate counterargument or by dropping the criticized belief
Maybe; I can’t say I’ve noticed that so much myself -- e.g. he just disappeared from this discussion when I refuted his assumptions about philosophy of language (that underpin his objection to zombies), but I haven’t seen him retract his claim that zombies are demonstrably incoherent.
e.g. he just disappeared from this discussion when I refuted his assumptions about philosophy of language (that underpin his objection to zombies), but I haven’t seen him retract his claim that zombies are demonstrably incoherent.
Clearly, from his standpoint a lot of things you believed were confused, and he decided against continuing to argue. This is a statement about willingness to engage situations where someone’s wrong on the Internet and presence of disagreement, not external evidence about correctness (distinct from your own estimate of correctness of your opponent’s position).
I won’t actually argue, just list some things that seem to be points where Richard talks past the intended meaning of the posts (irrespective of technical accuracy of the statements in themselves, if their meaning intended by Richard was what the posts referred to). Link to the post for convenience.
“premise that words refer to whatever generally causes us to utter them”: There is a particular sense of “refer” in which we can trace the causal history of words being uttered.
“It’s worth highlighting that this premise can’t be right, for we can talk about things that do not causally affect us. “: Yes, we can consider other senses of “refer”, make the discussion less precise, but those are not the senses used.
“We know perfectly well what we mean by the term ‘phenomenal consciousness’.”: Far from “perfectly well”.
“We most certainly do not just mean ‘whatever fills the role of causing me to make such-and-such utterances’” Maybe we don’t reason so, but it’s one tool to see what we actually mean, even if it explores this meaning in a different sense from what’s informally used (as a way of dissolving a potentially wrong question).
“No, the example of unicorns is merely to show that we can talk about non-causally related things.”: We can think/talk about ideas that cause us to think/talk about them in certain ways, and in this way the meaning of the idea (as set of properties which our minds see in it) causally influences uttering of words about it. Whether what the idea refers to causally influences us in other ways is irrelevant. On the other hand, if it’s claimed that the idea talks about the world (and is not an abstract logical fact unrelated to the world), there must be a pattern (event) of past observations that causes the idea to be evaluated as “correct”, and alternative observations that cause it to be evaluated as “wrong” (or a quantitative version of that). If that’s not possible, then it can’t be about our world.
Yes, if you’re talking about corporations, you cannot use exactly the same math than you do if you’re talking about evolutionary biology. But there are still some similarities that make it useful to know things about how selection works in evolutionary biology. Eliezer seems to be saying that if you want to call something “evolution”, then it has to meet these strictly-chosen criteria that he’ll tell you. But pretty much the only justification he offers is “if it doesn’t meet these criteria, then Price’s equation doesn’t apply”, and I don’t see why “evolution” would need to be strictly defined as “those processes which behave in a way specified by Price’s equation”. It can still be a useful analogy.
The rest are fine in my eyes, though the argument in The Psychological Unity of Humankind seems rather overstated for severalreasons.
Do you have recommendations for people/books that take this perspective seriously and then go on to explore interesting things with it? I haven’t seen anyone include the memetic perspective as part of their everyday worldview besides some folk at SIAI and yourself, which I find pretty sad.
Also, I get the impression you have off-kilter-compared-to-LW views on evolutionary biology, though I don’t remember any concrete examples. Do you have links to somewhere where I could learn more about what phenomena/perspectives you think aren’t emphasized or what not?
In academia, memetics is typically called “cultural evolution”. Probably the best book on that is “Not by Genes Alone”.
Your “evolutionary biology” question is rather vague. The nearest thing that springs to mind is this. Common views on that topic around here are more along the lines expressed in the The Robot’s Rebellion. If I am in a good mood, I describe such views as “lacking family values”—and if I am not, they get likened to a “culture of death”.
His response to it, or that it’s done so infrequently?
I for one am less worried the less often he writes things that are plainly false, so his being called out rarely doesn’t strike me as a cause for concern.
What scares me is that people say EY’s position is “plainly false” so rarely. Even if EY is almost always right, you would still expect a huge number of people to say that his positions are plainly false, especially when talking about such difficult and debated questions as those of philosophy and predicting the future.
What scares me is that people say EY’s position is “plainly false” so rarely.
What scares me is how often people express this concern relative to how often people actually agree with EY. Eliezer’s beliefs and assertions take an absolute hammering. I agree with him fairly often—no surprise, he is intelligent, has a similar cognitive style mine and has spent a whole lot of time thinking. But I disagree with him vocally whenever he seems wrong. I am far from the only person who does so.
If the topics are genuinely difficult, I don’t think it’s likely that many people who understand them would argue that Eliezer’s points are plainly false. Occasionally people drop in to argue such who clearly don’t have a very good understanding of rationality or the subject material. People do disagree with Eliezer for more substantive reasons with some frequency, but I don’t find the fact that they rarely pronounce him to be obviously wrong particularly worrying.
Most of the people who are most likely to think that EY’s positions on things are plainly false probably don’t bother registering here to say so.
There’s one IRC channel populated with smart CS / math majors, where I drop LW links every now and then. Pretty frequently they’re met with a rather critical reception, but while those people are happy to tear them apart on IRC, they have little reason to bother to come to LW and explain in detail why they disagree.
(Of the things they disagree on, I mainly recall that they consider Eliezer’s treatment of frequentism / Bayesianism as something of a strawman and that there’s no particular reason to paint them as two drastically differing camps when real statisticians are happy with using methods drawn from both.)
they consider Eliezer’s treatment of frequentism / Bayesianism as something of a strawman and that there’s no particular reason to paint them as two drastically differing camps when real statisticians are happy with using methods drawn from both.
In that case, we got very different impressions about how Eliezer described the two camps; here is what I heard: <channel righteous fury of Eliezer’s pure Bayesian soul>
It’s not Bayesian users on the one hand and Frequentists on the other, each despising the others’ methods. Rather, it’s the small group of epistemic statisticians and a large majority of instrumentalist ones.
The epistemics are the small band of AI researchers using statistical models to represent probability so as to design intelligence, learning, and autonomy. The idea is that ideal models are provably Baysian, and the task undertaken is to understand and implement close approximations of them.
The instrumentalist mainstream doesn’t always claim that it’s representing probability and doesn’t feel lost without that kind of philosophical underpinning. Instrumentalists hound whatever problem is at hand with all statistical models and variables that they can muster to get the curve or isolated variable etc. they’re looking for and think is best. The most important part of instrumentalist models is the statistician him or herself, which does the Bayesian updating adequately and without the need for understanding.
</channel righteous fury of Eliezer’s pure Bayesian soul>
Saying that the division is a straw man because most statisticians use all methods misses the point.
Most of the people who are most likely to think that EY’s positions on things are plainly false probably don’t bother registering here to say so.
True, but I still wouldn’t expect sharp disagreement with Eliezer to be so rare. One contributing factor may be that Eliezer at least appears to be so confident in so many of his positions, and does not put many words of uncertainty into his writing about theoretical issues.
When I first found this site, I read through all the OB posts chronologically, rather than reading the Sequences as sequences. So I got to see the history of several commenters, many of whom disagreed sharply with EY, with their disagreement evolving over several posts.
They tend to wander off after a while. Which is not surprising, as there is very little reward for it.
So I guess I’d ask this a different way: if you were an ethical philosopher whose positions disagreed with EY, what in this community would encourage you to post (or comment) about your disagreements?
if you were an ethical philosopher whose positions disagreed with EY, what in this community would encourage you to post (or comment) about your disagreements?
The presence of a large, sharp, and serious audience. The disadvantage, of course, is that the audience tends not to already be familiar with standard philosophical jargon.
By contrast, at a typical philosophy blog, you can share your ideas with an audience that already knows the jargon, and is also sharp and serious. The disadvantage, of course, is that at the typical philosophy blog, the audience is not large.
These considerations suggest that a philosopher might wish to produce his own competing meta-ethics sequence here if he were in the early stages of producing a semi-popular book on his ideas. He might be less interested if he is interested only in presenting to trained philosophers.
So I guess I’d ask this a different way: if you were an ethical philosopher whose positions disagreed with EY, what in this community would encourage you to post (or comment) about your disagreements?
Caring about the future of humanity, I suppose, and thinking that SIAI’s choices may have a big impact on the future of humanity. That’s what motivates me to post my disagreements—in an effort to figure out what’s correct.
Unfortunately, believing that the SIAI is likely to have a significant impact on the future of humanity already implies accepting many of its core claims: that the intelligence-explosion view of the singularity is accurate, that special measures to produce a friendly AI are feasible and necessary, and that the SIAI has enough good ideas that it has a reasonable chance of not being beaten to the punch by some other project. Otherwise it’s either chasing a fantasy or going after a real goal in a badly suboptimal way, and either way it’s not worth spending effort on influencing.
That still leaves plenty of room for disagreement with Eliezer, theoretically, but it narrows the search space enough that I’m not sure there are many talented ethical philosophers left in it whose views diverge significantly from the SIAI party line. There aren’t so many ethical philosophers in the world that a problem this specialized is going to attract very many of them.
As a corollary, I think it’s a good idea to downplay the SIAI applications of this site. Human rationality is a much broader topic than the kind of formal ethics that go into the SIAI’s work, and seems more likely to attract interesting and varied attention.
if you were an ethical philosopher whose positions disagreed with EY, what in this community would encourage you to post (or comment) about your disagreements?
I occasionally go through phases where it amuses me to be heckled about being a deontologist; does that count? (I was at one time studying for a PhD in philosophy and would likely have concentrated in ethics if I’d stayed.)
I suppose? Though if I generalize from that, it seems that the answer to luke’s original question is “because not enough people enjoy being heckled.” Which, um… well, I guess that’s true, in some sense.
It seems to me that EY himself addressed all three of the objections you list (though of course this doesn’t imply he addressed them adequately).
(1) It’s incompatible with the datum that substantive, fundamental normative disagreement is in fact possible. People may share the concept of a normative reason, even if they fundamentally disagree about which features of actions are the ones that give us reasons.
My own thinking is that humans tend to have the same underlying (evolved) structures behind our hard-to-articulate meta-ethical heuristics, even when we disagree broadly on object-level ethical issues (and of course hand-pick our articulations of the meta-criteria to support our object-level beliefs- the whole machinery of bias applies here).
This implies both that my object-level beliefs can be at odds with their meta-level criteria (if this becomes too obvious for me to rationalize away, I’m more likely to change one or other object-level belief than to change the meta-level heuristic), and that you and I can disagree fundamentally on the object level while still believing that there’s something in common which makes argumentation relevant to our disagreement.
OK. I’ll reply here because if I reply there, you won’t get the notifications.
The crux of your argument, it seems to me, is the following intuition:
Rather, it is essential to the concept of morality that it involves shared standards common to all fully reasonable agents.
This is certainly a property we would want morality to have, and one which human beings naturally assume it must have– but is that the central property of it? Should it turn out that nothing which looks like morality has this property, does it logically follow that all morality is dead, or is that reaction just a human impulse?
(I will note, with all the usual caveats, that believing one’s moral sentiments to be universal in scope and not based on preference is a big advantage in object-level moral arguments, and that we happen to be descended from the winners of arguments about tribal politics and morality.)
If a certain set of moral impulses involves shared standards common to, say, every sane human being, then moral arguments would still work among those human beings, in exactly the way you would want them to work across all intelligent beings. Frankly, that’s good enough for me. Why give baby-eating aliens in another universe veto powers over every moral intuition of yours?
Thanks for the reply—I find this a very interesting topic. One thing I should clarify is that my view doesn’t entail giving aliens “veto powers”, as you put it; an alternative response is to take them to be unreasonable to intrinsically desire the eating of babies. That isn’t an intrinsically desirable outcome (I take it), i.e. there is no reason to desire such a thing. Stronger still, we may think it intrinsically undesirable, so that insofar as an agent has such desires they are contrary to reason. (This requires a substantive notion of reason that goes beyond mere instrumental rationality, of course.)
In any case, I’d put the crux of my argument slightly differently. The core intuition is just that it’s possible to have irresolvable moral disagreements. We can imagine a case where Bob is stubbornly opposed to abortion, and Jane is just as stubbornly in favour of it, and neither agent is disposed to change their mind in light of any additional information. EY’s view would seem to imply that the two agents mustn’t really disagree. And that just seems a mistake: it’s part of our concept of morality that this very conceptcould be shared by someone who fundamentally (and irresolvably) disagrees with us about what the substantive moral facts are. This is because we’re aspiring to conform our judgments to a standard that is outside of ourselves. (If you don’t think there are any such objective standards, then that’s just to say that there are no normative facts, given my concept of normativity.)
Human beings are analogous to computers. Morality and other aspects of behavior and cognition are analogous to programs. It is a type error to ask whether a program “really exists” somewhere outside a computer, or is “intrinsic” to a computer, or is “contingent”, or something like that. Such questions don’t correspond to observations within the world that could turn out one way or the other. You see a computer running a certain program and that’s the end of the story.
Your mind is a program too, and your moral intuitions are how your algorithm feels from inside, not a direct perception of external reality (human beings are physically incapable of that kind of thing, though they may feel otherwise). I know for a fact that you have no astral gate in your head to pull answers from the mysterious source of morality. But this doesn’t imply that your moral intuitions “should” be worthless to you and you “should” seek external authority! There’s nothing wrong with mankind living by its internal moral lights.
Yes, it’s possible that different computers will have different programs. Our world contains billions of similar “moist robots” running similar programs, perhaps because we were all created from design documents that are 99% identical for historical reasons, and also because we influence each other a lot. Your intuition that all “possible” sentient agents must share a common morality is unlikely to survive an encounter with any sentient agent that’s substantially different from a human. We can imagine such agents easily, e.g. a machine that will search for proofs to Goldbach’s conjecture and turn surrounding matter and energy into computing machinery to that end. Such a machine may be more ingenious than any human in creating other machines, discovering new physics, etc., but will never gravitate toward your intuition that one shouldn’t kill babies. Most possible “intelligent agents” (aka algorithms that can hit small targets in large search spaces) aren’t humans in funny suits.
I expect Richard’s memeset has understanding of all your points that doesn’t move his current position. You’re probably exposing him to arguments he has already encountered, so there’s little point in expecting a different result. I’m not saying that Richard can’t be moved by argument, just not by standard argument that is already known to have failed to move him. He even probably “agrees” with a lot of your points, just with a different and more sophisticated understanding than yours.
On the other hand, it might work for the benefit of more naive onlookers.
The intent of my comment wasn’t to convince Richard (I never do that), but to sharpen our points and make him clarify whatever genuine insight he possesses and we don’t.
Your last paragraph suggests that you’ve misunderstood my view. I’m not making an empirical claim to the effect that all agents will eventually converge to our values—I agree that that’s obviously false. I don’t even think that all formally intelligent agents are guaranteed to have normative concepts like ‘ought’, ‘reason’, or ‘morality’. The claim is just that such a radically different agent could share our normative concepts (in particular, our aspiration to a mind-independent standard), even if they would radically disagree with us about which things fall under the concept. We could both have full empirical knowledge about our own and each other’s desires/dispositions, and yet one (or both) of us might be wrong about what we really have reason to want and to do.
(Aside: the further claim about “reasons” in your last sentence presupposes a subjectivist view about reasons that I reject.)
What use is this concept of “reasonability”? Let’s say I build an agent that wants to write the first 1000 Fibonacci numbers in mile-high digits on the Moon, except skipping the 137th one. When you start explaining to the agent that it’s an “arbitrary omission” and it “should” amend its desires for greater “consistency”, the agent just waves you off because listening to you isn’t likely to further its current goals. Listening to you is not rational for the agent in the sense that most people on LW use the term: it doesn’t increase expected utility. If by “rational” you mean something else, I’d like to understand what exactly.
I mean ‘rational’ in the ordinary, indefinable sense, whereby calling a decision ‘irrational’ expresses a distinctive kind of criticism—similar to that expressed by the words ‘crazy’, ‘foolish’, ‘unwise’, etc. (By contrast, you can just say “maximizes expected utility” if you really mean nothing more than maximizes expected utility—but note that that’s a merely descriptive concept, not a normative one.)
If you don’t possess this concept—if you never have thoughts about what’s rational, over and above just what maximizes expected utility—then I can’t help you.
When we’re trying to reduce intuitions, there’s no helping starting from informal ideas. Another question is that it’s not proper to stop there, but Richard doesn’t exactly suggest that.
A more salient to me question is, What are suggestions about redefining this “rational” intuitive idea good for, if it’s supposed to be the source material? Such question even explains how the idea of considering, say, “consciousness”, in a more precise sense is methodologically a step in the wrong direction: when words are the data you work with, you should be careful to assign new words to new ideas used for analyzing them.
I’m not sure I understand your second paragraph. Are you suggesting that if we come up with a new theory to explain some aspect of consciousness, we should use a word other than “consciousness” in that theory, to avoid potentially losing some of our intuitions about consciousness?
First, EY makes it abundantlyclear that two agents can have a fundamental disagreement on values– it’s just not the best (or most helpful) assumption when you’re talking about two sane human beings with a vast sea of common frameworks and heuristics.
Secondly, I’m worried about what you’re trying to do with words when you suggest we “take them to be unreasonable to intrinsically desire the eating of babies”.
If you’re making an empirical claim that an alien with fundamentally different terminal values will (say) be uninterested in negotiating mutually beneficial deals, or will make patently suboptimal decisions by its own criteria, or exhibit some other characteristic of what we mean by “unreasonable”, then you’d need some strong evidence for that claim.
If instead you openly redefine “reasonable” to include “shares our fundamental moral standards”, then the property
it is essential to the concept of morality that it involves shared standards common to all fully reasonable agents
becomes a tautology which no longer excludes “meta-semantic subjectivism”, as you put it. So I’m puzzled what you mean.
Talking past each other a bit here. Let me try again.
EY makes it abundantly clear that two agents can have a fundamental disagreement on values
EY allows for disagreement in attitude: you might want one thing, while the babyeaters want something different. Of course I’m not charging him with being unable to accommodate this. The objection is instead that he’s unable to accommodate disagreement in moral judgment (at the fundamental level). Normativity as mere semantics, and all that.
Your second point rests on a false dichotomy. I’m not making an empirical claim, but nor am I merely defining the word “reasonable”. Rather, I’m making a substantive normative (non-empirical) hypothesis about which things are reasonable. If you can’t make sense of the idea of a substantive non-empirical issue, you may have fallen victim to scientism.
The core intuition is just that it’s possible to have irresolvable moral disagreements.
What is the difference between an in-principle irresolvable disagreement (moral or otherwise), and talking past each other (i.e. talking of different subject matters, or from different argument-processing frameworks)?
an alternative response is to take them to be unreasonable to intrinsically desire the eating of babies
What fact have you established by manipulating the definition of a word in this manner? I want a meta-ethical theory that at least describes baby-eaters, because I don’t expect to have object-level understanding of human morality that is substantially more accurate than what you’d get if you add baby-eating impulses to it.
This summary of Eliezer’s position seems to ignore the central part about computation. That is, Eliezer does not say that ‘Right’ means ‘promotes external goods X, Y and Z’ but rather that it means a specific computation that can be roughly characterized as ‘renormalizing intuition’
I see the project of morality as a project of renormalizing intuition. We have intuitions about things that seem desirable or undesirable, intuitions about actions that are right or wrong, intuitions about how to resolve conflicting intuitions, intuitions about how to systematize specific intuitions into general principles.
which eventually outputs something like ‘promotes external goods X, Y and Z’. I think Eliezer would argue that at least some of the objections list here are not valid if we add the part about computation. (Specifically, disagreements and fallibility can result from from lack of logical omniscience regarding the output of the ‘morality’ computation.)
Is the reason for skipping over this part of Eliezer’s idea that standard (Montague) semantic theory treats all logically equivalent language as having the same intension? (I believe this is known as “the logical omniscience problem” in linguistics and philosophy of language.)
The part about computation doesn’t change the fundamental structure of the theory. It’s true that it creates more room for superficial disagreement and fallibility (of similar status to disagreements and fallibility regarding the effective means to some shared terminal values), but I see this as an improvement in degree and not in kind. It still doesn’t allow for fundamental disagreement and fallibility, e.g. amongst logically omniscient agents.
(I take it to be a metaethical datum that even people with different terminal values, or different Eliezerian “computations”, can share the concept of a normative reason, and sincerely disagree about which (if either) of their values/computations is correctly tracking the normative reasons. Similarly, we can coherently doubt whether even our coherently-extrapolated volitions would be on the right track or not.)
It still doesn’t allow for fundamental disagreement and fallibility, e.g. amongst logically omniscient agents.
It’s not clear to me why there must be fundamental disagreement and fallibility, e.g. amongst logically omniscient agents. Can you refer me to an argument or intuition pump that explains why you think that?
One related argument is the Open Question Argument: for any natural property F that an action might have, be it promotes my terminal values, or is the output of an Eliezerian computation that models my coherent extrapolated volition, or whatever the details might be, it’s always coherent to ask: “I agree that this action is F, but is it good?”
But the intuitions that any metaethics worthy of the name must allow for fundamental disagreement and fallibility are perhaps more basic than this. I’d say they’re just the criteria that we (at least, many of us) have in mind when insisting that any morality worthy of the name must be “objective”, in a certain sense. These two criteria are proposed as capturing that sense of objectivity that we have in mind. (Again, don’t you find something bizarrely subjectivist about the idea that we’re fundamentally morally infallible—that we can’t even question whether our fundamental values / CEV are really on the right track?)
I’d say they’re just the criteria that we (at least, many of us) have in mind when insisting that any morality worthy of the name must be “objective”, in a certain sense.
What would you say to someone who does not share your intuition that such “objective” morality likely exists?
My main problem with objective morality is that while it’s hard to deny that there seem to be mind-independent moral facts like “pain is morally bad”, there doesn’t seem to be enough such facts to build an ethical system out of them. What natural phenomena count as pain, exactly? How do we trade off between pain and pleasure? How do we trade off between pain in one person, and annoyance in many others? How do we trade off pain across time (i.e., should we discount future pain, if so how)? Across possible worlds? How do we morally treat identical copies? It seems really hard, perhaps impossible, to answer these questions without using subjective preferences or intuitions that vary from person to person, or worse, just picking arbitrary answers when we don’t even have any relevant preferences or intuitions. If it turns out that such subjectivity and/or arbitrariness can’t be avoided, that would be hard to square with objective morality actually existing.
(Again, don’t you find something bizarrely subjectivist about the idea that we’re fundamentally morally infallible—that we can’t even question whether our fundamental values / CEV are really on the right track?)
I do think there’s something wrong with saying that we can’t question whether CEV is really on the right track. But I wouldn’t use the words “bizarrely subjectivist”. To me the problem is just that I clearly can and do question whether CEV is really on the right track. Fixing this seems to require retreating quite a bit from Eliezer’s metaethical position (but perhaps there is some other solution that I’m not thinking of). At this point I would personally take the following (minimalist) position:
At least some people, at least some of the time, refer to the same concept by “morality” as me and they have substantive disagreements over its nature and content.
I’m not confident about any of its properties.
Running CEV (if it were practical to) seems like a good way to learn more about the nature and content of morality, but there may be (probably are) better ways.
If it turns out that such subjectivity and/or arbitrariness can’t be avoided, that would be hard to square with objective morality actually existing.
Compare with formal systems giving first-order theories of standard model of natural numbers. You can’t specify the whole thing, and at some point you run into (independent of what comes before) statements for which it’s hard to decide whether they hold for the standard naturals, and so you could add to the theory either those statements or their negation. Does this break the intuition that there is some intended structure corresponding to natural numbers, or more pragmatically that we can still usefully seek better theories that capture it? For me, it doesn’t in any obvious way.
It seems to be an argument in favor of arithmetic being objective that almost everyone agree that a certain a set of axioms correctly characterize what natural numbers are (even if incompletely), and from that set of axioms we can derive much (even if not all) of what we want to know about the properties of natural numbers. If arithmetic were in the same situation as morality is today, it would be much harder (i.e., more counterintuitive) to claim that (1) everyone is referring to the same thing by “arithmetic” and “natural numbers” and (2) arithmetic truths are mind-independent.
To put it another way, conditional on objective morality existing, you’d expect the situation to be closer to that of arithmetic. Conditional on it not existing, you’d expect the situation to be closer to what it actually is.
What would you say to someone who does not share your intuition that such “objective” morality likely exists?
I’d say: be an error theorist! If you don’t think objective morality exists, then you don’t think that morality exists. That’s a perfectly respectable position. You can still agree with me about what it would take for morality to really exist. You just don’t think that our world actually has what it takes.
Yes, that makes sense, except that my intuition that objective morality does not exist is not particularly strong either. I guess what I was really asking was, do you have any arguments to the effect that objective morality exists?
(2) The semantic tricks merely shift the lump under the rug, they don’t get rid of it. Standard worries about relativism re-emerge, e.g. an agent can know a priori that their own fundamental values are right, given how the meaning of the word ‘right’ is determined. This kind of (even merely ‘fundamental’) infallibility seems implausible.
EY bites this bullet in the abstract, but notes that it does not apply to humans. An AI with a simple utility function and full ability to analyze its own source code can be quite sure that maximizing that function is the meaning of “that-AI-right” in the sense EY is talking about.
But there is no analogue to that situation in human psychology, given how much we now know about self-deception, our conscious and unconscious mental machinery, and the increasing complexity of our values the more we think on them. We can, it’s true, say that “the correct extrapolation of my fundamental values is what’s right for me to do”, but this doesn’t guarantee whether value X is or is not a member of that set. The actual work of extrapolating human values (through moral arguments and other methods) still has to be done.
So practical objections to this sort of bullet-biting don’t apply to this metaethics; are there any important theoretical objections?
EDIT: Changed “right” to “that-AI-right”. Important clarification.
An AI with a simple utility function and full ability to analyze its own source code can be quite sure that maximizing that function is the meaning of “that-AI-right” in the sense EY is talking about.
I don’t think that’s right, or EY’s position (I’d like evidence on that). Who’s to say that maximization is precisely what’s right? That might be a very good heuristic, but upon reflection the AI might decide to self-improve in a way that changes this subgoal (of the overall decision problem that includes all the other decision-making parts), by finding considerations that distinguish maximizing attitude to utility and the right attitude to utility. It would of course use its current utility-maximizing algorithm to come to that decision. But the conclusion might be that too much maximization is bad for environment or something. The AI would stop maximizing for the reason it’s not the most maximizing thing, the same way as a person would not kill for the reason that action leads to a death, even though avoid-causing-death is not the whole morality and doesn’t apply universally.
Agreed that on EY’s view (and my own), human “fundamental values” (1) have not yet been fully articulated/extrapolated; that we can’t say with confidence whether X is in that set.
But AFAICT, EY rejects the idea (which you seem here to claim that he endorses?) that an AI with a simple utility function can be sure that maximizing that function is the right thing to do. It might believe that maximizing that function is the right thing to do, but it would be wrong. (2)
AFAICT this is precisely what RichardChappell considers implausible: the idea that unlike the AI, humans can correctly believe that maximizing their utility function is the right thing to do.
==
(1) Supposing there exist any such things, of which I am not convinced.
(2) Necessarily wrong, in fact, since on EY’s view as I understand it there’s one and only one right set of values, and humans currently implement it, and the set of values humans implement is irreducably complex and therefore cannot be captured by a simple utility function. Therefore, an AI maximizing a simple utility function is necessarily not doing the right thing on EY’s view.
Sorry, I meant to use the two-place version; it wouldn’t be what’s right; what I meant is that the completely analogous concept of “that-AI-right” would consist simply of that utility function.
To the extent that you are still talking about EY’s views, I still don’t think that’s correct… I think he would reject the idea that “that-AI-right” is analogous to right, or that “right” is a 2-place predicate.
That said, given that this question has come up elsethread and I’m apparently in the minority, and given that I don’t understand what all this talk of right adds to the discussion in the first place, it becomes increasingly likely that I’ve just misunderstood something.
In any case, I suspect we all agree that the AI’s decisions are motivated by its simple utility function in a manner analogous to how human decisions are motivated by our (far more complex) utility function. What disagreement exists, if any, involves the talk of “right” that I’m happy to discard altogether.
(1): I think it’s a prominent naturalistic feature; as EY said above, in a physical universe there are only quantum amplitudes, and if two agents have sufficiently accurate knowledge about the physical configuration of something, including their respective minds, they have to agree about that configuration, regardless of that they possibly have different values.
(2): I’m personally a bit confused about Eliezer’s constant promotion of a language that de-subjectivizes morality. In most debates “objective” and “subjective” may entail a confusion when viewed in a naturalistic light; however, as I understand Eliezer’s stance does boil down to a traditionally subjective viewpoint in the sense that it opposes the religious notion of morality as light shining down from the skies (and the notion of universally compelling arguments).
In regards to infallibility, an agent at most times has imperfect knowledge of right; I can’t see how subjectivity entails infallibility. I don’t even have perfect access to my current values, and there is also a huge set of moral arguments that would compel me to modify my current values if I heard them.
(3) The “why right means promoting X and Y” question is addressed by a recursive justification as discussed here and very specifically in the last paragraphs of Meaning of Right. If I ask “why should I do what is right?”, that roughly means “why should I do what I should do?” or “why is right what is right?”. I happen to be a mind that is compelled by a certain class of moral arguments, and I can reflect on this fact using my current mind, and, naturally, find that I’m compelled by a certain class of moral arguments.
re: infallibility—right, the objection is not that you could infallibly know that XYZ is right. Rather, the problem is that you could infallibly know that your fundamental values are right (though you might not know what your fundamental values are).
Rephrased, this knowledge is just the notion that you instantiate some computation instead of not doing (or being) anything. This way, my confidence in its truth is very high, although of course not 1.
We know we instantiate some computation. But it’s a pre-theoretic datum that we don’t know that our fundamental values are right. So EY’s theory misdescribes the concept of rightness.
(This is basically a variation on Moore’s Open Question Argument.)
I’d be okay with a strong AI that correctly followed my values, regardless of whether they’re “right” by any other criterion.
If you think you wouldn’t be okay with such an AI, I suspect the most likely explanation is that you’re confused about the concept of “your values”. Namely, if you yearn to discover some simple external formula like the categorical imperative and then enact the outcomes prescribed by that formula, then that’s just another fact about your personal makeup that has to be taken into account by the AI.
And if you agree that you would be okay with such an AI, that means Eliezer’s metaethics is adequate for its stated goal (creating friendly AI), whatever other theoretical drawbacks it might have.
What is objection (1) saying? That asserting there are moral facts is incompatible with the fact that people disagree about what they are? Specifically, when people agree that there is such a thing as a reason that applies to both of them, they disagree about how the reason is caused by reality?
Do we not then say they are both wrong about there being one “reason”?
I speak English(LD). You speak English(RC). The difference between our languages is of the same character as that between a speaker of Spanish and a speaker of French. I say “I” and you correctly read it as referring to lessdazed. You say “I” and I correctly read it as referring to RichardChapell. I have reasons(LD). You have reasons(RC). Do you think that were we perfect at monitoring what we each meant when we said anything and knew the relevant consequences of actions, the two of us would be capable of disagreeing when one of us asserted something in a sentence using the word “moral”? Why?
That asserting there are moral facts is incompatible with the fact that people disagree about what they are?
No, I think there are moral facts and that people disagree about what they are. But such substantive disagreement is incompatible with Eliezer’s reductive view on which the very meaning of ‘morality’ differs from person to person. It treats ‘morality’ like an indexical (e.g. “I”, “here”, “now”), which obviously doesn’t allow for real disagreement.
Compare: “I am tall.” “No, I am not tall!” Such an exchange would be absurd—the people are clearly just talking past each other, since there is no common referent for ‘I’. But moral language doesn’t plausibly function like this. It’s perfectly sensible for one person to say, “I ought to have an abortion”, and another to disagree: “No, you ought not to have an abortion”. (Even if both are logically omniscient.) They aren’t talking past each other. Rather, they’re disagreeing about the morality of abortion.
But moral language doesn’t plausibly function like this.
It’s not plausible(RC, 7/1/2011 4:25 GMT), but it is plausible(LD, 7/1/2011 4:25 GMT).
Compare: “I am tall.” “No, I am not tall!” Such an exchange would be absurd—the people are clearly just talking past each other, since there is no common referent for ‘I’.
It’s not impossible for people to be confused in exactly such a way.
It’s perfectly sensible for one person to say, “I ought to have an abortion”, and another to disagree: “No, you ought not to have an abortion”. (Even if both are logically omniscient.)
That’s begging the question.
That intuition pump imagines intelligent people disagreeing, finds it plausible, notices that intelligent people disagreeing proves nothing, then replaces the label “intelligent” with “omniscient” (since that, if proven, would prove something) without showing the work that would make the replacement valid. If the work could be shown, the intuition pump wouldn’t be very valuable, as one could just use the shown work for persuasion rather than the thought experiment with the disagreeing people. I strongly suspect that the reason the shown work is unavailable is because it does not exist.
Eliezer’s reductive view on which the very meaning of ‘morality’ differs from person to person.
Forget morality for one second. Doesn’t the meaning of the word “hat” differ from person to person?
It’s perfectly sensible for one person to say, “I ought to have an abortion”
It’s only sensible to say if/because context forestalls equivocation (or tries to, anyway). Retroactively removing the context by coming in the conversation with a different meaning of ought (even if the first meaning of “ought” was “objective values, as I think they are, as I think I want them to be, that are universally binding on all possible minds, and I would maintain under any coherent extrapolation of my values” where the first person is wrong about those facts and the second meaning of “ought” is the first person’s extrapolated volition) introduces equivocation. It’s really analogous to saying “No, I am not tall”.
Where the first person says “X would make me happy, I want to feel like doing X, and others will be better off according to balancing equation Y if I do X, and the word “ought” encompasses when those things coincide according to objective English, so I ought to do X”, and the second person says “X would make you happy, you want to feel like doing X, and others will not be better off according to balancing equation Z if you do X, and the word “ought” encompasses when those things coincide according to objective English, so you ought not do X”, they are talking past each other. Purported debates about the true meaning of “ought” reveal that everyone has their own balancing equation, and the average person thinks all others are morally obliged by objective morality to follow his or her equation. In truth, the terms “make happy” and want to feel like doing” are rolled up into the balancing equation, but in it (for Westerners) terms for self and others seem as if they are of different kind.
Purported debates about the true meaning of “ought” reveal that everyone has their own balancing equation, and the average person thinks all others are morally obliged by objective morality to follow his or her equation.
You’re confusing metaethics and first-order ethics. Ordinary moral debates aren’t about the meaning of “ought”. They’re about the first-order question of which actions have the property of being what we ought to do. People disagree about which actions have this property. They posit different systematic theories (or ‘balancing equations’ as you put it) as a hypothesis about which actions have the property. They aren’t stipulatively defining the meaning of ‘ought’, or else their claim that “You ought to follow the prescriptions of balancing equation Y” would be tautological, rather than a substantive claim as it is obviously meant to be.
Ordinary moral debates aren’t about the meaning of “ought”.
I know that, which is why I said “Purported debates about the true meaning of ‘ought’” rather than “ordinary debates, which are about the true meaning of ‘ought’”.
They’re about the first-order question of which actions have the property of being what we ought to do. People disagree about which actions have this property.
Please be careful not to beg the question. People agree that there is such a property, but that is something about which they can be wrong.
They aren’t stipulatively defining the meaning of ‘ought’, or else their claim that “You ought to follow the prescriptions of balancing equation Y” would be tautological...
Rather, they aren’t trying to stipulatively define the meaning of ‘ought’, or else their claim that “You ought to follow the prescriptions of balancing equation Y” would be tautological.
In fact, due to people’s poor self-insight, time limits, and the sometimes over-coarse granularity of language, they do not stipulate their actual balancing equation. Had they perfect insight and ability to represent their insights, it would be such a tautology. They would cease to speak like that had they the additional insight that for it to do the work it is called upon to do,”ought” is a word that needs grounding in the context of the real reasons for action of beings More generally, they are speaking an idiolect even regarding other definitions.
...rather than a substantive claim as it is obviously meant to be.
It’s meant to be such a claim, but it is in error because the speaker is confused about morality, and in a sense is not even wrong. They are claiming some actions have an objective moral valuation binding upon all intelligent beings, but they may as well claim the action has the property of being a square circle—or better yet, a perfect circle for pi is exactly 3, which is something I have witnessed a religious person claim is true.
~~~~~~~~~
I don’t understand either why you believe as you do or what good justification you might have for it.
I can see why one might want to make truth claims in which it falls out the folk have the least amount of confusion to be embarrassed about and are least wrong, and if one begins with the assumption that there are “moral facts” in the strongest sense, that’s a good start. However, that neither prevents one from having to say they are wrong about an enormous amount nor does it prevent one from having to claim that others, such as me, are wrong about moral facts. I can also see how it might be comforting were moral facts real in that sense and how the world might be better off if people believe in them, but obviously those things would not make them true.
At least you seem to understand the position you are arguing against and I am advocating. You repeatedly have tried to dissuade me from it by pointing out extensions of my position and counting on me to disclaim it from reductio ad absurdum. But it always seems to be by appealing to the same notion, that people simply cannot be wrong about some sorts of things they believe, which is a position I reject.
So, assuming I bite the bullet and don’t find it implausible that some people are fundamentally confused when they make moral claims, what other reasons might there be to agree with them? Why should I believe that intensionally defined “objective morality” has any extension at all?
I’m not arguing for moral realism here. I’m arguing against metaethical reductionism, which leaves open either realism OR error theory.
For all I’ve said, people may well be mistaken when they attribute normative properties to things. That’s fine. I’m just trying to clarify what it is that people are claiming when they make moral claims. This is conceptual analysis, not metaphysics. I’m pointing out that what you claim to be the meaning of ‘morality’ isn’t what people mean to be talking about when they engage in moral discourse. I’m not presupposing that ordinary people have any great insight into the nature of reality, but they surely do have some idea of what their own words mean. Your contrary linguistic hypothesis seems completely unfounded.
No, you learned that the tooth fairy doesn’t exist, and that your mother was instead responsible for the observable phenomena that you had previously attributed to the tooth fairy.
(It’s a good analogy though. I do think that claiming that morality exists “as a computation” is a lot like claiming that the tooth fairy really exists “as one’s mother”.)
No, you learned that the tooth fairy doesn’t exist
No.
Usually, the first thing to do when guessing about a random number from 1-100 is to split the possibilities in half by asking if it is more than 50 (or odd. Etc.)
The tooth fairy example gets a variety of responses, from people insisting it is just objectively wrong to say “the tooth fairy doesn’t exist” to those saying it is just objectively wrong to say the tooth fairy was really my mother. I happen to agree with you about what the best way is to describe what went on in this specific case. However, this is a standard blegg-rube situation that is unusual only in that it is not clear which way is best to describe the phenomenon to others.
There is a constellation of phenomena that correlate to each other—the fairy being female, being magic, having diaphanous wings, collecting things for money, those things being stored under pillows, those things being teeth. None of these is qualitatively essential to be a tooth fairy to most people than “having ten fingers” is essential to being human. If tomorrow we learn that magic is real, a female sprite collects teeth from under pillows, and does so on the back of a termite (and has size-changing technology/magic, why not?), most people would naively say “the tooth fairy does not fly, but burrows on the back of a termite”. That’s OK, but not great if the true nature of the situation is not recognized, and they fall into error if they think “tooth fairy” has a meaning divorced from flight. Likewise, those who say “there was never a ‘tooth fairy’, there is rather the ‘burrowing tooth fairy’” are right that there was never a thing exactly like the classic description, but this group makes an error if they demand the first stop calling the “burrowing tooth fairy” the “tooth fairy”.
There is more to say, an individual who makes up explanations ad hoc is not communicating, and the relative confluence of idiolects is valid because of the tinkerbell effect. that makes saying “No, you learned that the tooth fairy doesn’t exist” really peculiar, in the face of the many who hold the opposite position, likewise endorsed as language by common usage!
When it comes to “morality”, the constellation of things represented with a token labeled “second order morality” had several stars, significantly organized around a few (mutually contradictory, but all validly denoted by the word “morality”) sets of answers to these eight questions in the SEP on Moral Reasoning:
How do relevant considerations get taken up in moral reasoning?
Is it essential to moral reasoning for the considerations it takes up to be crystallized into, or ranged under, principles?
How do we sort out which moral considerations are most relevant?
In what ways do motivational elements shape moral reasoning?
What is the best way to model the kinds of conflicts among considerations that arise in moral reasoning?
Does moral reasoning include learning from experience and changing one’s mind?
How can we reason, morally, with one another?
What is the importance of institutional authority and social roles to the structure of moral reasoning?
One characteristic of things often (and validly) called “morality” is that they include the following among other pieces to answer the above questions: “‘That which is good’ is an ephemeral property that is infused in things in a dualistic way, it can be separate from all concerns beings may have, “good” can be binding in a way that is tautological and pointless...etc.
Reductionism continues the tradition of “morality”, like other words, not meaning exactly what it did in the past, and it may be better to think of it as error theory in some cases. But all new theories claim the ones they supersede are in error, and there is nothing special about “morality minus magical deontology” or “moral reductionism” that one should consider them bad English by the usual standard.
After typing all of this, it strikes me that cars are a better example of no one component being essential. Cars without wheels will be “flying cars”, unless they get sufficient lift and air etc. such that as one goes down the line of models from flying jalopy to sports flying car, each successive flying car will be called “car” by fewer and fewer people as it more resembles a fighter jet or so...as perhaps our descendants will call nuclear-powered flying vehicles.
The important thing is that reductionism gets it right when describing reality, the problems with deontology etc., and explains why people think erroneously, and it then proceeds to tell one how to act such that first order moral claims have meaning. If despite all that, you want to focus on its erosion of folk theories to call it an error theory, that is understandable, but it will be uninteresting if you say to the multitude “It is wrong(RC) to call reductionism ‘moral’(RC),” since it is not wrong(everybody who recognizes not to misuse language by fixating on a connotation that often went along with a word when the contents described by that word are otherwise vindicated and under discussion, and we’re discussing language which means 1) constelations and 2) tinkerbell).
The tooth fairy example gets a variety of responses
Seriously? I’ve never heard anyone insist that the tooth fairy really exists (in the form of their mother). It would seem most contrary to common usage (in my community, at least) to use ‘Tooth Fairy’ to denote “whoever replaced the tooth under my pillow with a coin”. The magical element is (in my experience) treated as essential to the term and not a mere “connotation”.
I’ve heard of the saying you mention, but I think you misunderstand people when you interpret it literally. My response was not intended as some “peculiar” declaration of mind-independent meaning facts, but rather as a straightforward interpretation of what people who utter such claims have in mind when they do so. (Ask them, “Do you mean that the tooth fairy exists?” and I expect the response, “No, silly, I just mean that my mother is responsible for the coin under my pillow.”)
So, to clarify: I don’t think that there are free-floating “meaning” facts out there independently of our linguistic dispositions. I just dispute whether your definitions adequately capture the things that most people really care about (i.e. treat as essential) when using the terms in question.
It’s no excuse to say that metaethical reductionism “gets reality right” when the whole dispute is instead over whether they have accommodated (or rather eliminated) some concept of which we have a pre-theoretic grasp. Compare the theological reductionist thesis that “God is love”. Love exists, therefore God exists, voila! If someone pointed out that this view is needlessly misleading since love is not what most people mean to be talking about when they speak of ‘God’ (and it would be more honest to just admit one’s atheism), it would be no response to give a lecture about constellations and tinkerbell.
I just dispute whether your definitions adequately capture the things that most people really care about (i.e. treat as essential) when using the terms in question.
It’s no excuse to say that metaethical reductionism “gets reality right” when the whole dispute is instead over whether they have accommodated (or rather eliminated) some concept of which we have a pre-theoretic grasp.
What if metaethical reductionism is not meant (by some) to accommodate the pre-theoretic grasp of “morality” of most people, but just to accommodate the pre-theoretic grasp of “morality” of people like lessdazed? Could metaethical reductionism be considered a “respectable position” in that sense?
And separately, suppose the main reason I’m interested in metaethics is that I am trying to answer a question like “Should I terminally value the lives of random strangers?” and I’m not sure what that question means exactly or how I should go about answering it. In this case, is there a reason for me to care much about the pre-theoretic grasp of most people, as opposed to, say, people I think are most likely to be right about morality?
Compare the theological reductionist thesis that “God is love”. Love exists, therefore God exists, voila!
This is a good example, as people saying this are in some ways doing the opposite of what you advocate, but in other ways, they are doing the same thing.
I think people are motivated to say “God is love” out of a desire to avoid being logically compelled to view certain others as wrong (there is a twist, some passive aggressive non-believers say it to claim atheists are wrong when saying theists are wrong). The exact motivation isn’t important, but its presence would provide a countervailing force against the sheer silliness of their confusion (which is worse than yours in an important way) and explain how they could possibly make it.
The mistake being made is to pretend that there are inherently attached meanings to a word, as if those words simply had that meaning, regardless of context of using words in general, and that word in particular, when that context is totally adequate to explain meaning. When it is clear that contemporary theists and all their ancestors were in error, the hippie says “God is love” to pretend there is no disagreement, and that “God” meant love—instead of what was obviously meant, or often weirdly in addition to the contradictory thing that they admit was meant.
You do the opposite in that rather than seek ways to interpret others as agreeing or being right, even when there is obvious disagreement, you seek ways to interpret others as disagreeing or being wrong. You use the exact same mechanism of ignoring context and how language is used.
You avoid the worst error of the hippies, which is claiming that others mean both “God is love” and “God is an agent, etc.”
However, reductionists take the history of language and find words have many connotations, the meaning of “moral” and find it has a long history of meaning many different things, many of several things, and that the meaning people accept as associated to the term has to do with the quantity and quality of many aspects, none of which are intrinsic.
You have apparently decided that it’s magic or moral facts are the type of facts that can’t be explained in terms of how the world is is the element absolutely essential to “morality”, when (presumably, as so many random hundreds I’ve encountered accept the reductionist usage) thousands of others do not.
I doubt that were there a room full of hippies totally separated from theism that they would feel any need to obfuscate by saying “God is love”, which is why it is appropriate to barge in and correct the usage, as the usage is confusion designed to create the illusion of agreement.
You are attempting to create an illusion of disagreement about morality, where your disagreement is with how multitudes use “morality”—and that is not the sort of argument that it is possible to win because if that’s how we’re using it, that makes it right when we talk without confusion among each other, and reasonably use it with those whose linguistic sensibilities we don’t know explicitly. The reason why we’re doing so is because it most clearly communicates (out of all the words we could have used), and our stretching the meaning of “morality” to exclude magic is no more unusual than the stretching of the term “write” to include jabbing at keys with the fingers. We would continue to use it so in absence of non-reductionists, whereas hippies would revert to “love” in absence of theism.
I’ve heard of the saying you mention, but I think you misunderstand people when you interpret it literally.
Please understand, the disagreement is not about what happens but about what it means to literally be the tooth fairy.
That depends whether you are going to replace objective morality with subective morality or with error theory. It can be argued that subjectivism is an unstable position that amount to error theory.
(3) Just as simple subjectivism is an implausible theory of what ‘right’ means, so Eliezer’s meta-semantic subjectivism is an implausible theory of why ‘right’ means promoting external goods X, Y, Z. An adequately objective metaethics shouldn’t even give preferences a reference-fixing role.
This seems to me like begging the question. Can you expand on this?
Thinking more about this, it may have been better if Eliezer had not framed his meta-ethics sequence around “the meaning of right.”
If we play rationalist’s taboo with our moral terms and thus avoid moral terms altogether, what Eliezer seems to be arguing is that what we really care about is not (a) that whatever states of affairs our brains are wired to send reward signals in response to be realized, but (b) that we experience peace and love and harmony and discovery and so on.
His motivation for thinking this way is a thought experiment—which might become real in the relatively near future—about what would happen if a superintelligent machine could rewire our brains. If what we really care about is (a), then we shouldn’t object if the superintelligent machine rewires our brains to send reward signals only when we are sitting in a jar. But we would object to that scenario. Thus, what we care about seems not to be (a) but (b).
In a meta-ethicists terms, we could interpret Eliezer not as making an argument about the meaning of moral terms, but instead as making an argument that (b) is what gives us Reasons, not (a).
Now, all this meta-babble might not matter much. I’m pretty sure even if I was persuaded that the correct meta-ethical theory states that I should be okay with releasing a superintelligence that would rewire me to enjoy sitting in a jar, I would do whatever I could to prevent such a scenario and instead promote a superintelligence that would bring peace and joy and harmony and discovery and so on.
This is a cool formulation. It’s interesting that there are other things that can happen to you not similar to “being persuaded of a metaethical theory” that entail that whenever you are told to do X you’re compelled to do X. (Voodoo or whatever.)
I could get into how much I hate this kind of rejoinder if you bait me some more. I wasn’t asking you for the number of acres in a square mile. Let me just rephrase:
I hadn’t heard of motivational internalism before, could you expand your comment?
what Eliezer seems to be arguing is that what we really care about is not (a) that whatever states of affairs our brains are wired to send reward signals in response to be realized, but (b) that we experience peace and love and harmony and discovery and so on.
His motivation for thinking this way is a thought experiment—which might become real in the relatively near future—about what would happen if a superintelligent machine could rewire our brains. If what we really care about is (a), then we shouldn’t object if the superintelligent machine rewires our brains to send reward signals only when we are sitting in a jar.
I don’t see what plausible reasoning process could lead you to infer this unlikely statement (about motivation, given how many detail would need to be just right for the statement to happen to be true).
Also, even if you forbid modifying definition of human brain, things that initiate high-reward signals in our brains (or that we actually classify as “harmony” or “love”) are very far from what we care about, just as whatever a calculator actually computes is not the same kind of consideration as the logically correct answer, even if you use a good calculator and aren’t allowed sabotage. There are many reasons (and contexts) for reward in human brain to not be treated as indicative of goodness of a situation.
It was an explanation for why your thought experiment provides a bad motivation: we can just forbid modification of human brains to stop the thought experiment from getting through, but that would still leave a lot of problems, which shows that just this thought experiment is not sufficient motivation.
Sure, the superintelligence thought experiment is not the fully story.
One problem with the suggestion of writing a rule to not alter human brains comes in specifying how the machine is not allowed to alter human brains. I’m skeptical about our ability to specify that rule in a way that does not lead to disastrous consequences. After all, our brains are being modified all the time by the environment, by causes that are on a wide spectrum of ‘direct’ and ‘indirect.’
Other problems with adding such a rule are given here.
Come on, this tiny detail isn’t worth the discussion. Classical solution to wireheading, asking the original and not the one under the influence, referring to you-at-certain-time and not just you-concept that resolves to something unpredicted at any given future time in any given possible world, rigid-designator-in-time.
Eliezer’s metaethics might be clarified in terms of the distinctions between sense, reference, and reference-fixing descriptions. I take it that Eliezer wants to use ‘right’ as a rigid designator to denote some particular set of terminal values, but this reference fact is fixed by means of a seemingly ‘relative’ procedure (namely, whatever terminal values the speaker happens to hold, on some appropriate [if somewhat mysterious] idealization). Confusions arise when people mistakenly read this metasemantic subjectivism into the first-order semantics or meaning of ‘right’.
In summary:
(i) ‘Right’ means, roughly, ‘promotes external goods X, Y and Z’
(ii) claim i above is true because I desire X, Y, and Z.
Note that Speakers Use Their Actual Language, so murder would still be wrong even if I had the desires of a serial killer. But if I had those violent terminal values, I would speak a slightly different language than I do right now, so that when KillerRichard asserts “Murder is right!” what he says is true. We don’t really disagree, but are instead merely talking past each other.
Virtues of the theory:
(a) By rigidifying on our actual, current desires (or idealizations thereupon), it avoids Inducing Desire Satisfactions.
(b) Shifting the subjectivity out to the metasemantic level leaves us with a first-order semantic proposal that at least does a better job than simple subjectivism at ‘saving the phenomena’. (It has echoes of Mark Schroeder’s desire-based view of reasons, according to which the facts that give us reasons are the propositional contents of our desires, rather than the desires themselves. Or something like that.)
(c) It’s naturalistic, if you find moral non-naturalism ‘spooky’. (Though I’d sooner recommend Mackie-style error theory for naturalists, since I don’t think (b) above is enough to save the phenomena.)
Objections
(1) It’s incompatible with the datum that substantive, fundamental normative disagreement is in fact possible. People may share the concept of a normative reason, even if they fundamentally disagree about which features of actions are the ones that give us reasons.
(2) The semantic tricks merely shift the lump under the rug, they don’t get rid of it. Standard worries about relativism re-emerge, e.g. an agent can know a priori that their own fundamental values are right, given how the meaning of the word ‘right’ is determined. This kind of (even merely ‘fundamental’) infallibility seems implausible.
(3) Just as simple subjectivism is an implausible theory of what ‘right’ means, so Eliezer’s meta-semantic subjectivism is an implausible theory of why ‘right’ means promoting external goods X, Y, Z. An adequately objective metaethics shouldn’t even give preferences a reference-fixing role.
I think this is an excellent summary. I would make the following comments:
Yes, but I think Eliezer was mistaken in identifying this kind of confusion as the fundamental source of the objections to his theory (as in the Löb’s theorem discussion). Sophisticated readers of LW (or OB, at the time) are surely capable of distinguishing between logical levels. At least, I am—but nevertheless, I still didn’t feel that his theory was adequately “non-relativist” to satisfy the kinds of people who worry about “relativism”. What I had in mind, in other words, was your objections (2) and (3).
The answer to those objections, by the way, is that an “adequately objective” metaethics is impossible: the minds of complex agents (such as humans) are the only place in the universe where information about morality is to be found, and there are plenty of possible minds in mind-design space (paperclippers, pebblesorters, etc.) from which it is impossible to extract the same information. This directly answers (3), anyway; as for (2), “fallibility” is rescued (on the object level) by means of imperfect introspective knowledge: an agent could be mistaken about what its own terminal values are.
Note that your answer to (2) also answers (1): value uncertainty makes it seem as if there is substantive, fundamental normative disagreement even if there isn’t. (Or maybe there is if you don’t buy that particular element of EY’s theory)
Elizer attempted to deal with that problem by defining a certain set of things as “h-right”, that is, morally right from the frame of reference of the human mind. He made clear that alien entities probably would not care about what is h-right, but that humans do, and that’s good enough.
That’s not a reason to prefer EY’s theory to an error theory (according to which properly normative properties would have to be irreducibly normative, but no such properties actually exist).
Richard,
Until persuaded otherwise, I agree with you on this point. (These days, I take Richard Joyce to have the clearest defense of error theory, and I just subtract his confusing-to-me defense of fictionalism.) Besides, I think there are better ways of getting something like an ‘objective’ ethical theory (in something like a ‘realist’ sense) while still holding that reasons for action arise only from desires, or from relations between desires and states of affairs. In fact, that’s the kind of theory I defend: desirism. Though, I’m not too interested anymore in whether desirism is to be called ‘objective’ or ‘realist’, even though I think a good case can be made for both.
Richard,
You’re speaking my language, thanks! I hope this is EY’s view, because I know what this means. Maybe now I can go back and read EY’s sequence in light of this interpretation and it will make more sense to me now.
EY’s theory as presented above makes me suspicious that making basic evaluative moral terms rigid designators is a kind of ‘trick’ which, though perhaps not intended, very easily has the effect of carrying along some common absolutist connotations of those terms where they no longer apply in EY’s use of those terms.
At the moment, I’m not so worried about objection (1), but objections (2) and (3) are close to what bother me about EY’s theory, especially if this is foundational for EY’s thinking about how we ought to be designing a Friendly AI. If we’re working on a project as important as Friendly AI, it becomes an urgent problem to get our meta-ethics right, and I’m not sure Eliezer has done it yet. Which is why we need more minds working on this problem. I hope to be one of those minds, even if my current meta-ethics turns out to be wrong (I’ve held my current meta-ethics for under 2 years, anyway, and it has shifted slightly since adoption).
But, at the moment it remains plausible to me that Eliezer is right, and I just don’t see why right now. Eliezer is a very smart guy who has invested a lot of energy into training himself to think straight about things and respond to criticism either with adequate counterargument or by dropping the criticized belief.
Maybe; I can’t say I’ve noticed that so much myself -- e.g. he just disappeared from this discussion when I refuted his assumptions about philosophy of language (that underpin his objection to zombies), but I haven’t seen him retract his claim that zombies are demonstrably incoherent.
Clearly, from his standpoint a lot of things you believed were confused, and he decided against continuing to argue. This is a statement about willingness to engage situations where someone’s wrong on the Internet and presence of disagreement, not external evidence about correctness (distinct from your own estimate of correctness of your opponent’s position).
You think that “clearly” Eliezer believed many of Richard’s beliefs were confused. Which beliefs, do you think?
I won’t actually argue, just list some things that seem to be points where Richard talks past the intended meaning of the posts (irrespective of technical accuracy of the statements in themselves, if their meaning intended by Richard was what the posts referred to). Link to the post for convenience.
“premise that words refer to whatever generally causes us to utter them”: There is a particular sense of “refer” in which we can trace the causal history of words being uttered.
“It’s worth highlighting that this premise can’t be right, for we can talk about things that do not causally affect us. “: Yes, we can consider other senses of “refer”, make the discussion less precise, but those are not the senses used.
“We know perfectly well what we mean by the term ‘phenomenal consciousness’.”: Far from “perfectly well”.
“We most certainly do not just mean ‘whatever fills the role of causing me to make such-and-such utterances’” Maybe we don’t reason so, but it’s one tool to see what we actually mean, even if it explores this meaning in a different sense from what’s informally used (as a way of dissolving a potentially wrong question).
“No, the example of unicorns is merely to show that we can talk about non-causally related things.”: We can think/talk about ideas that cause us to think/talk about them in certain ways, and in this way the meaning of the idea (as set of properties which our minds see in it) causally influences uttering of words about it. Whether what the idea refers to causally influences us in other ways is irrelevant. On the other hand, if it’s claimed that the idea talks about the world (and is not an abstract logical fact unrelated to the world), there must be a pattern (event) of past observations that causes the idea to be evaluated as “correct”, and alternative observations that cause it to be evaluated as “wrong” (or a quantitative version of that). If that’s not possible, then it can’t be about our world.
This is the first time I saw anyone telling EY that what he wrote is plainly false.
I made a personal list of top frequently-cited-yet-irritatingly-misleading EY posts:
http://lesswrong.com/lw/l6/no_evolutions_for_corporations_or_nanodevices/
http://lesswrong.com/lw/iv/the_futility_of_emergence/
http://lesswrong.com/lw/l0/adaptationexecuters_not_fitnessmaximizers/
http://lesswrong.com/lw/rl/the_psychological_unity_of_humankind/
http://lesswrong.com/lw/y3/value_is_fragile/
I agree with the first one of those being bad.
Yes, if you’re talking about corporations, you cannot use exactly the same math than you do if you’re talking about evolutionary biology. But there are still some similarities that make it useful to know things about how selection works in evolutionary biology. Eliezer seems to be saying that if you want to call something “evolution”, then it has to meet these strictly-chosen criteria that he’ll tell you. But pretty much the only justification he offers is “if it doesn’t meet these criteria, then Price’s equation doesn’t apply”, and I don’t see why “evolution” would need to be strictly defined as “those processes which behave in a way specified by Price’s equation”. It can still be a useful analogy.
The rest are fine in my eyes, though the argument in The Psychological Unity of Humankind seems rather overstated for several reasons.
FWIW, cultural evolution is not an analogy. Culture literally evolves—via differential reproductive success of memes...
Do you have recommendations for people/books that take this perspective seriously and then go on to explore interesting things with it? I haven’t seen anyone include the memetic perspective as part of their everyday worldview besides some folk at SIAI and yourself, which I find pretty sad.
Also, I get the impression you have off-kilter-compared-to-LW views on evolutionary biology, though I don’t remember any concrete examples. Do you have links to somewhere where I could learn more about what phenomena/perspectives you think aren’t emphasized or what not?
My current project is a book on memetics. I also have a blog on memetics.
Probably the best existing book on the topic is The Meme Machine by Susan Blackmore.
I also maintain some memetics links, some memetics references, a memetics glossary—and I have a bunch of memetics videos.
In academia, memetics is typically called “cultural evolution”. Probably the best book on that is “Not by Genes Alone”.
Your “evolutionary biology” question is rather vague. The nearest thing that springs to mind is this. Common views on that topic around here are more along the lines expressed in the The Robot’s Rebellion. If I am in a good mood, I describe such views as “lacking family values”—and if I am not, they get likened to a “culture of death”.
Wow, thanks! Glad I asked. I will start a tab explosion.
Really? That’s kind of scary...
His response to it, or that it’s done so infrequently?
I for one am less worried the less often he writes things that are plainly false, so his being called out rarely doesn’t strike me as a cause for concern.
What scares me is that people say EY’s position is “plainly false” so rarely. Even if EY is almost always right, you would still expect a huge number of people to say that his positions are plainly false, especially when talking about such difficult and debated questions as those of philosophy and predicting the future.
What scares me is how often people express this concern relative to how often people actually agree with EY. Eliezer’s beliefs and assertions take an absolute hammering. I agree with him fairly often—no surprise, he is intelligent, has a similar cognitive style mine and has spent a whole lot of time thinking. But I disagree with him vocally whenever he seems wrong. I am far from the only person who does so.
If the topics are genuinely difficult, I don’t think it’s likely that many people who understand them would argue that Eliezer’s points are plainly false. Occasionally people drop in to argue such who clearly don’t have a very good understanding of rationality or the subject material. People do disagree with Eliezer for more substantive reasons with some frequency, but I don’t find the fact that they rarely pronounce him to be obviously wrong particularly worrying.
Most of the people who are most likely to think that EY’s positions on things are plainly false probably don’t bother registering here to say so.
There’s one IRC channel populated with smart CS / math majors, where I drop LW links every now and then. Pretty frequently they’re met with a rather critical reception, but while those people are happy to tear them apart on IRC, they have little reason to bother to come to LW and explain in detail why they disagree.
(Of the things they disagree on, I mainly recall that they consider Eliezer’s treatment of frequentism / Bayesianism as something of a strawman and that there’s no particular reason to paint them as two drastically differing camps when real statisticians are happy with using methods drawn from both.)
In that case, we got very different impressions about how Eliezer described the two camps; here is what I heard: <channel righteous fury of Eliezer’s pure Bayesian soul>
It’s not Bayesian users on the one hand and Frequentists on the other, each despising the others’ methods. Rather, it’s the small group of epistemic statisticians and a large majority of instrumentalist ones.
The epistemics are the small band of AI researchers using statistical models to represent probability so as to design intelligence, learning, and autonomy. The idea is that ideal models are provably Baysian, and the task undertaken is to understand and implement close approximations of them.
The instrumentalist mainstream doesn’t always claim that it’s representing probability and doesn’t feel lost without that kind of philosophical underpinning. Instrumentalists hound whatever problem is at hand with all statistical models and variables that they can muster to get the curve or isolated variable etc. they’re looking for and think is best. The most important part of instrumentalist models is the statistician him or herself, which does the Bayesian updating adequately and without the need for understanding. </channel righteous fury of Eliezer’s pure Bayesian soul>
Saying that the division is a straw man because most statisticians use all methods misses the point.
Edit: see for example here and here.
True, but I still wouldn’t expect sharp disagreement with Eliezer to be so rare. One contributing factor may be that Eliezer at least appears to be so confident in so many of his positions, and does not put many words of uncertainty into his writing about theoretical issues.
When I first found this site, I read through all the OB posts chronologically, rather than reading the Sequences as sequences. So I got to see the history of several commenters, many of whom disagreed sharply with EY, with their disagreement evolving over several posts.
They tend to wander off after a while. Which is not surprising, as there is very little reward for it.
So I guess I’d ask this a different way: if you were an ethical philosopher whose positions disagreed with EY, what in this community would encourage you to post (or comment) about your disagreements?
The presence of a large, sharp, and serious audience. The disadvantage, of course, is that the audience tends not to already be familiar with standard philosophical jargon.
By contrast, at a typical philosophy blog, you can share your ideas with an audience that already knows the jargon, and is also sharp and serious. The disadvantage, of course, is that at the typical philosophy blog, the audience is not large.
These considerations suggest that a philosopher might wish to produce his own competing meta-ethics sequence here if he were in the early stages of producing a semi-popular book on his ideas. He might be less interested if he is interested only in presenting to trained philosophers.
(nods) That makes sense.
Caring about the future of humanity, I suppose, and thinking that SIAI’s choices may have a big impact on the future of humanity. That’s what motivates me to post my disagreements—in an effort to figure out what’s correct.
Unfortunately, believing that the SIAI is likely to have a significant impact on the future of humanity already implies accepting many of its core claims: that the intelligence-explosion view of the singularity is accurate, that special measures to produce a friendly AI are feasible and necessary, and that the SIAI has enough good ideas that it has a reasonable chance of not being beaten to the punch by some other project. Otherwise it’s either chasing a fantasy or going after a real goal in a badly suboptimal way, and either way it’s not worth spending effort on influencing.
That still leaves plenty of room for disagreement with Eliezer, theoretically, but it narrows the search space enough that I’m not sure there are many talented ethical philosophers left in it whose views diverge significantly from the SIAI party line. There aren’t so many ethical philosophers in the world that a problem this specialized is going to attract very many of them.
As a corollary, I think it’s a good idea to downplay the SIAI applications of this site. Human rationality is a much broader topic than the kind of formal ethics that go into the SIAI’s work, and seems more likely to attract interesting and varied attention.
I’m curious if among those you saw left, there were any who you wish had stayed.
It seems to me that if one has a a deep need for precise right answers, it’s hard to beat participating in the LW community.
Not especially; the disagreements never seem to resolve.
I occasionally go through phases where it amuses me to be heckled about being a deontologist; does that count? (I was at one time studying for a PhD in philosophy and would likely have concentrated in ethics if I’d stayed.)
I suppose? Though if I generalize from that, it seems that the answer to luke’s original question is “because not enough people enjoy being heckled.” Which, um… well, I guess that’s true, in some sense.
It seems to me that EY himself addressed all three of the objections you list (though of course this doesn’t imply he addressed them adequately).
Moral Error and Moral Disagreement confronts this.
My own thinking is that humans tend to have the same underlying (evolved) structures behind our hard-to-articulate meta-ethical heuristics, even when we disagree broadly on object-level ethical issues (and of course hand-pick our articulations of the meta-criteria to support our object-level beliefs- the whole machinery of bias applies here).
This implies both that my object-level beliefs can be at odds with their meta-level criteria (if this becomes too obvious for me to rationalize away, I’m more likely to change one or other object-level belief than to change the meta-level heuristic), and that you and I can disagree fundamentally on the object level while still believing that there’s something in common which makes argumentation relevant to our disagreement.
Yeah, I’m the “Richard4” in the comments thread there :-)
OK. I’ll reply here because if I reply there, you won’t get the notifications.
The crux of your argument, it seems to me, is the following intuition:
This is certainly a property we would want morality to have, and one which human beings naturally assume it must have– but is that the central property of it? Should it turn out that nothing which looks like morality has this property, does it logically follow that all morality is dead, or is that reaction just a human impulse?
(I will note, with all the usual caveats, that believing one’s moral sentiments to be universal in scope and not based on preference is a big advantage in object-level moral arguments, and that we happen to be descended from the winners of arguments about tribal politics and morality.)
If a certain set of moral impulses involves shared standards common to, say, every sane human being, then moral arguments would still work among those human beings, in exactly the way you would want them to work across all intelligent beings. Frankly, that’s good enough for me. Why give baby-eating aliens in another universe veto powers over every moral intuition of yours?
Thanks for the reply—I find this a very interesting topic. One thing I should clarify is that my view doesn’t entail giving aliens “veto powers”, as you put it; an alternative response is to take them to be unreasonable to intrinsically desire the eating of babies. That isn’t an intrinsically desirable outcome (I take it), i.e. there is no reason to desire such a thing. Stronger still, we may think it intrinsically undesirable, so that insofar as an agent has such desires they are contrary to reason. (This requires a substantive notion of reason that goes beyond mere instrumental rationality, of course.)
In any case, I’d put the crux of my argument slightly differently. The core intuition is just that it’s possible to have irresolvable moral disagreements. We can imagine a case where Bob is stubbornly opposed to abortion, and Jane is just as stubbornly in favour of it, and neither agent is disposed to change their mind in light of any additional information. EY’s view would seem to imply that the two agents mustn’t really disagree. And that just seems a mistake: it’s part of our concept of morality that this very concept could be shared by someone who fundamentally (and irresolvably) disagrees with us about what the substantive moral facts are. This is because we’re aspiring to conform our judgments to a standard that is outside of ourselves. (If you don’t think there are any such objective standards, then that’s just to say that there are no normative facts, given my concept of normativity.)
Richard, hello.
Human beings are analogous to computers. Morality and other aspects of behavior and cognition are analogous to programs. It is a type error to ask whether a program “really exists” somewhere outside a computer, or is “intrinsic” to a computer, or is “contingent”, or something like that. Such questions don’t correspond to observations within the world that could turn out one way or the other. You see a computer running a certain program and that’s the end of the story.
Your mind is a program too, and your moral intuitions are how your algorithm feels from inside, not a direct perception of external reality (human beings are physically incapable of that kind of thing, though they may feel otherwise). I know for a fact that you have no astral gate in your head to pull answers from the mysterious source of morality. But this doesn’t imply that your moral intuitions “should” be worthless to you and you “should” seek external authority! There’s nothing wrong with mankind living by its internal moral lights.
Yes, it’s possible that different computers will have different programs. Our world contains billions of similar “moist robots” running similar programs, perhaps because we were all created from design documents that are 99% identical for historical reasons, and also because we influence each other a lot. Your intuition that all “possible” sentient agents must share a common morality is unlikely to survive an encounter with any sentient agent that’s substantially different from a human. We can imagine such agents easily, e.g. a machine that will search for proofs to Goldbach’s conjecture and turn surrounding matter and energy into computing machinery to that end. Such a machine may be more ingenious than any human in creating other machines, discovering new physics, etc., but will never gravitate toward your intuition that one shouldn’t kill babies. Most possible “intelligent agents” (aka algorithms that can hit small targets in large search spaces) aren’t humans in funny suits.
I expect Richard’s memeset has understanding of all your points that doesn’t move his current position. You’re probably exposing him to arguments he has already encountered, so there’s little point in expecting a different result. I’m not saying that Richard can’t be moved by argument, just not by standard argument that is already known to have failed to move him. He even probably “agrees” with a lot of your points, just with a different and more sophisticated understanding than yours.
On the other hand, it might work for the benefit of more naive onlookers.
The intent of my comment wasn’t to convince Richard (I never do that), but to sharpen our points and make him clarify whatever genuine insight he possesses and we don’t.
That’s a motivation I didn’t consider. (Agreed.)
Yeah, as Vladimir guessed, this is all familiar.
Your last paragraph suggests that you’ve misunderstood my view. I’m not making an empirical claim to the effect that all agents will eventually converge to our values—I agree that that’s obviously false. I don’t even think that all formally intelligent agents are guaranteed to have normative concepts like ‘ought’, ‘reason’, or ‘morality’. The claim is just that such a radically different agent could share our normative concepts (in particular, our aspiration to a mind-independent standard), even if they would radically disagree with us about which things fall under the concept. We could both have full empirical knowledge about our own and each other’s desires/dispositions, and yet one (or both) of us might be wrong about what we really have reason to want and to do.
(Aside: the further claim about “reasons” in your last sentence presupposes a subjectivist view about reasons that I reject.)
What use is this concept of “reasonability”? Let’s say I build an agent that wants to write the first 1000 Fibonacci numbers in mile-high digits on the Moon, except skipping the 137th one. When you start explaining to the agent that it’s an “arbitrary omission” and it “should” amend its desires for greater “consistency”, the agent just waves you off because listening to you isn’t likely to further its current goals. Listening to you is not rational for the agent in the sense that most people on LW use the term: it doesn’t increase expected utility. If by “rational” you mean something else, I’d like to understand what exactly.
I mean ‘rational’ in the ordinary, indefinable sense, whereby calling a decision ‘irrational’ expresses a distinctive kind of criticism—similar to that expressed by the words ‘crazy’, ‘foolish’, ‘unwise’, etc. (By contrast, you can just say “maximizes expected utility” if you really mean nothing more than maximizes expected utility—but note that that’s a merely descriptive concept, not a normative one.)
If you don’t possess this concept—if you never have thoughts about what’s rational, over and above just what maximizes expected utility—then I can’t help you.
I don’t think we can make progress with such imprecise thinking. Eliezer has a nice post about that.
When we’re trying to reduce intuitions, there’s no helping starting from informal ideas. Another question is that it’s not proper to stop there, but Richard doesn’t exactly suggest that.
A more salient to me question is, What are suggestions about redefining this “rational” intuitive idea good for, if it’s supposed to be the source material? Such question even explains how the idea of considering, say, “consciousness”, in a more precise sense is methodologically a step in the wrong direction: when words are the data you work with, you should be careful to assign new words to new ideas used for analyzing them.
In this old comment, he does seem to suggest stopping there.
I’m not sure I understand your second paragraph. Are you suggesting that if we come up with a new theory to explain some aspect of consciousness, we should use a word other than “consciousness” in that theory, to avoid potentially losing some of our intuitions about consciousness?
Yes.
First, EY makes it abundantly clear that two agents can have a fundamental disagreement on values– it’s just not the best (or most helpful) assumption when you’re talking about two sane human beings with a vast sea of common frameworks and heuristics.
Secondly, I’m worried about what you’re trying to do with words when you suggest we “take them to be unreasonable to intrinsically desire the eating of babies”.
If you’re making an empirical claim that an alien with fundamentally different terminal values will (say) be uninterested in negotiating mutually beneficial deals, or will make patently suboptimal decisions by its own criteria, or exhibit some other characteristic of what we mean by “unreasonable”, then you’d need some strong evidence for that claim.
If instead you openly redefine “reasonable” to include “shares our fundamental moral standards”, then the property
becomes a tautology which no longer excludes “meta-semantic subjectivism”, as you put it. So I’m puzzled what you mean.
Talking past each other a bit here. Let me try again.
EY allows for disagreement in attitude: you might want one thing, while the babyeaters want something different. Of course I’m not charging him with being unable to accommodate this. The objection is instead that he’s unable to accommodate disagreement in moral judgment (at the fundamental level). Normativity as mere semantics, and all that.
Your second point rests on a false dichotomy. I’m not making an empirical claim, but nor am I merely defining the word “reasonable”. Rather, I’m making a substantive normative (non-empirical) hypothesis about which things are reasonable. If you can’t make sense of the idea of a substantive non-empirical issue, you may have fallen victim to scientism.
What is the difference between an in-principle irresolvable disagreement (moral or otherwise), and talking past each other (i.e. talking of different subject matters, or from different argument-processing frameworks)?
What fact have you established by manipulating the definition of a word in this manner? I want a meta-ethical theory that at least describes baby-eaters, because I don’t expect to have object-level understanding of human morality that is substantially more accurate than what you’d get if you add baby-eating impulses to it.
Ah! Sorry for carrying coals to Newcastle, then. Let me catch up in that thread.
Yes, this is what I thought EY’s theory was. EY? Is this your view?
This summary of Eliezer’s position seems to ignore the central part about computation. That is, Eliezer does not say that ‘Right’ means ‘promotes external goods X, Y and Z’ but rather that it means a specific computation that can be roughly characterized as ‘renormalizing intuition’
which eventually outputs something like ‘promotes external goods X, Y and Z’. I think Eliezer would argue that at least some of the objections list here are not valid if we add the part about computation. (Specifically, disagreements and fallibility can result from from lack of logical omniscience regarding the output of the ‘morality’ computation.)
Is the reason for skipping over this part of Eliezer’s idea that standard (Montague) semantic theory treats all logically equivalent language as having the same intension? (I believe this is known as “the logical omniscience problem” in linguistics and philosophy of language.)
The part about computation doesn’t change the fundamental structure of the theory. It’s true that it creates more room for superficial disagreement and fallibility (of similar status to disagreements and fallibility regarding the effective means to some shared terminal values), but I see this as an improvement in degree and not in kind. It still doesn’t allow for fundamental disagreement and fallibility, e.g. amongst logically omniscient agents.
(I take it to be a metaethical datum that even people with different terminal values, or different Eliezerian “computations”, can share the concept of a normative reason, and sincerely disagree about which (if either) of their values/computations is correctly tracking the normative reasons. Similarly, we can coherently doubt whether even our coherently-extrapolated volitions would be on the right track or not.)
It’s not clear to me why there must be fundamental disagreement and fallibility, e.g. amongst logically omniscient agents. Can you refer me to an argument or intuition pump that explains why you think that?
One related argument is the Open Question Argument: for any natural property F that an action might have, be it promotes my terminal values, or is the output of an Eliezerian computation that models my coherent extrapolated volition, or whatever the details might be, it’s always coherent to ask: “I agree that this action is F, but is it good?”
But the intuitions that any metaethics worthy of the name must allow for fundamental disagreement and fallibility are perhaps more basic than this. I’d say they’re just the criteria that we (at least, many of us) have in mind when insisting that any morality worthy of the name must be “objective”, in a certain sense. These two criteria are proposed as capturing that sense of objectivity that we have in mind. (Again, don’t you find something bizarrely subjectivist about the idea that we’re fundamentally morally infallible—that we can’t even question whether our fundamental values / CEV are really on the right track?)
What would you say to someone who does not share your intuition that such “objective” morality likely exists?
My main problem with objective morality is that while it’s hard to deny that there seem to be mind-independent moral facts like “pain is morally bad”, there doesn’t seem to be enough such facts to build an ethical system out of them. What natural phenomena count as pain, exactly? How do we trade off between pain and pleasure? How do we trade off between pain in one person, and annoyance in many others? How do we trade off pain across time (i.e., should we discount future pain, if so how)? Across possible worlds? How do we morally treat identical copies? It seems really hard, perhaps impossible, to answer these questions without using subjective preferences or intuitions that vary from person to person, or worse, just picking arbitrary answers when we don’t even have any relevant preferences or intuitions. If it turns out that such subjectivity and/or arbitrariness can’t be avoided, that would be hard to square with objective morality actually existing.
I do think there’s something wrong with saying that we can’t question whether CEV is really on the right track. But I wouldn’t use the words “bizarrely subjectivist”. To me the problem is just that I clearly can and do question whether CEV is really on the right track. Fixing this seems to require retreating quite a bit from Eliezer’s metaethical position (but perhaps there is some other solution that I’m not thinking of). At this point I would personally take the following (minimalist) position:
At least some people, at least some of the time, refer to the same concept by “morality” as me and they have substantive disagreements over its nature and content.
I’m not confident about any of its properties.
Running CEV (if it were practical to) seems like a good way to learn more about the nature and content of morality, but there may be (probably are) better ways.
Compare with formal systems giving first-order theories of standard model of natural numbers. You can’t specify the whole thing, and at some point you run into (independent of what comes before) statements for which it’s hard to decide whether they hold for the standard naturals, and so you could add to the theory either those statements or their negation. Does this break the intuition that there is some intended structure corresponding to natural numbers, or more pragmatically that we can still usefully seek better theories that capture it? For me, it doesn’t in any obvious way.
It seems to be an argument in favor of arithmetic being objective that almost everyone agree that a certain a set of axioms correctly characterize what natural numbers are (even if incompletely), and from that set of axioms we can derive much (even if not all) of what we want to know about the properties of natural numbers. If arithmetic were in the same situation as morality is today, it would be much harder (i.e., more counterintuitive) to claim that (1) everyone is referring to the same thing by “arithmetic” and “natural numbers” and (2) arithmetic truths are mind-independent.
To put it another way, conditional on objective morality existing, you’d expect the situation to be closer to that of arithmetic. Conditional on it not existing, you’d expect the situation to be closer to what it actually is.
I’d say: be an error theorist! If you don’t think objective morality exists, then you don’t think that morality exists. That’s a perfectly respectable position. You can still agree with me about what it would take for morality to really exist. You just don’t think that our world actually has what it takes.
Yes, that makes sense, except that my intuition that objective morality does not exist is not particularly strong either. I guess what I was really asking was, do you have any arguments to the effect that objective morality exists?
EY bites this bullet in the abstract, but notes that it does not apply to humans. An AI with a simple utility function and full ability to analyze its own source code can be quite sure that maximizing that function is the meaning of “that-AI-right” in the sense EY is talking about.
But there is no analogue to that situation in human psychology, given how much we now know about self-deception, our conscious and unconscious mental machinery, and the increasing complexity of our values the more we think on them. We can, it’s true, say that “the correct extrapolation of my fundamental values is what’s right for me to do”, but this doesn’t guarantee whether value X is or is not a member of that set. The actual work of extrapolating human values (through moral arguments and other methods) still has to be done.
So practical objections to this sort of bullet-biting don’t apply to this metaethics; are there any important theoretical objections?
EDIT: Changed “right” to “that-AI-right”. Important clarification.
I don’t think that’s right, or EY’s position (I’d like evidence on that). Who’s to say that maximization is precisely what’s right? That might be a very good heuristic, but upon reflection the AI might decide to self-improve in a way that changes this subgoal (of the overall decision problem that includes all the other decision-making parts), by finding considerations that distinguish maximizing attitude to utility and the right attitude to utility. It would of course use its current utility-maximizing algorithm to come to that decision. But the conclusion might be that too much maximization is bad for environment or something. The AI would stop maximizing for the reason it’s not the most maximizing thing, the same way as a person would not kill for the reason that action leads to a death, even though avoid-causing-death is not the whole morality and doesn’t apply universally.
See also this comment.
Agreed that on EY’s view (and my own), human “fundamental values” (1) have not yet been fully articulated/extrapolated; that we can’t say with confidence whether X is in that set.
But AFAICT, EY rejects the idea (which you seem here to claim that he endorses?) that an AI with a simple utility function can be sure that maximizing that function is the right thing to do. It might believe that maximizing that function is the right thing to do, but it would be wrong. (2)
AFAICT this is precisely what RichardChappell considers implausible: the idea that unlike the AI, humans can correctly believe that maximizing their utility function is the right thing to do.
==
(1) Supposing there exist any such things, of which I am not convinced.
(2) Necessarily wrong, in fact, since on EY’s view as I understand it there’s one and only one right set of values, and humans currently implement it, and the set of values humans implement is irreducably complex and therefore cannot be captured by a simple utility function. Therefore, an AI maximizing a simple utility function is necessarily not doing the right thing on EY’s view.
Sorry, I meant to use the two-place version; it wouldn’t be what’s right; what I meant is that the completely analogous concept of “that-AI-right” would consist simply of that utility function.
To the extent that you are still talking about EY’s views, I still don’t think that’s correct… I think he would reject the idea that “that-AI-right” is analogous to right, or that “right” is a 2-place predicate.
That said, given that this question has come up elsethread and I’m apparently in the minority, and given that I don’t understand what all this talk of right adds to the discussion in the first place, it becomes increasingly likely that I’ve just misunderstood something.
In any case, I suspect we all agree that the AI’s decisions are motivated by its simple utility function in a manner analogous to how human decisions are motivated by our (far more complex) utility function. What disagreement exists, if any, involves the talk of “right” that I’m happy to discard altogether.
Thanks, Richard, for putting so much effort into your comment! When I find the time to parse this, I’ll come back here to comment.
(1): I think it’s a prominent naturalistic feature; as EY said above, in a physical universe there are only quantum amplitudes, and if two agents have sufficiently accurate knowledge about the physical configuration of something, including their respective minds, they have to agree about that configuration, regardless of that they possibly have different values.
(2): I’m personally a bit confused about Eliezer’s constant promotion of a language that de-subjectivizes morality. In most debates “objective” and “subjective” may entail a confusion when viewed in a naturalistic light; however, as I understand Eliezer’s stance does boil down to a traditionally subjective viewpoint in the sense that it opposes the religious notion of morality as light shining down from the skies (and the notion of universally compelling arguments).
In regards to infallibility, an agent at most times has imperfect knowledge of right; I can’t see how subjectivity entails infallibility. I don’t even have perfect access to my current values, and there is also a huge set of moral arguments that would compel me to modify my current values if I heard them.
(3) The “why right means promoting X and Y” question is addressed by a recursive justification as discussed here and very specifically in the last paragraphs of Meaning of Right. If I ask “why should I do what is right?”, that roughly means “why should I do what I should do?” or “why is right what is right?”. I happen to be a mind that is compelled by a certain class of moral arguments, and I can reflect on this fact using my current mind, and, naturally, find that I’m compelled by a certain class of moral arguments.
EDIT: see also komponisto’s comment.
re: infallibility—right, the objection is not that you could infallibly know that XYZ is right. Rather, the problem is that you could infallibly know that your fundamental values are right (though you might not know what your fundamental values are).
Rephrased, this knowledge is just the notion that you instantiate some computation instead of not doing (or being) anything. This way, my confidence in its truth is very high, although of course not 1.
We know we instantiate some computation. But it’s a pre-theoretic datum that we don’t know that our fundamental values are right. So EY’s theory misdescribes the concept of rightness.
(This is basically a variation on Moore’s Open Question Argument.)
Huh?
I’d be okay with a strong AI that correctly followed my values, regardless of whether they’re “right” by any other criterion.
If you think you wouldn’t be okay with such an AI, I suspect the most likely explanation is that you’re confused about the concept of “your values”. Namely, if you yearn to discover some simple external formula like the categorical imperative and then enact the outcomes prescribed by that formula, then that’s just another fact about your personal makeup that has to be taken into account by the AI.
And if you agree that you would be okay with such an AI, that means Eliezer’s metaethics is adequate for its stated goal (creating friendly AI), whatever other theoretical drawbacks it might have.
What is objection (1) saying? That asserting there are moral facts is incompatible with the fact that people disagree about what they are? Specifically, when people agree that there is such a thing as a reason that applies to both of them, they disagree about how the reason is caused by reality?
Do we not then say they are both wrong about there being one “reason”?
I speak English(LD). You speak English(RC). The difference between our languages is of the same character as that between a speaker of Spanish and a speaker of French. I say “I” and you correctly read it as referring to lessdazed. You say “I” and I correctly read it as referring to RichardChapell. I have reasons(LD). You have reasons(RC). Do you think that were we perfect at monitoring what we each meant when we said anything and knew the relevant consequences of actions, the two of us would be capable of disagreeing when one of us asserted something in a sentence using the word “moral”? Why?
Or have I misread things?
No, I think there are moral facts and that people disagree about what they are. But such substantive disagreement is incompatible with Eliezer’s reductive view on which the very meaning of ‘morality’ differs from person to person. It treats ‘morality’ like an indexical (e.g. “I”, “here”, “now”), which obviously doesn’t allow for real disagreement.
Compare: “I am tall.” “No, I am not tall!” Such an exchange would be absurd—the people are clearly just talking past each other, since there is no common referent for ‘I’. But moral language doesn’t plausibly function like this. It’s perfectly sensible for one person to say, “I ought to have an abortion”, and another to disagree: “No, you ought not to have an abortion”. (Even if both are logically omniscient.) They aren’t talking past each other. Rather, they’re disagreeing about the morality of abortion.
It’s not plausible(RC, 7/1/2011 4:25 GMT), but it is plausible(LD, 7/1/2011 4:25 GMT).
It’s not impossible for people to be confused in exactly such a way.
That’s begging the question.
That intuition pump imagines intelligent people disagreeing, finds it plausible, notices that intelligent people disagreeing proves nothing, then replaces the label “intelligent” with “omniscient” (since that, if proven, would prove something) without showing the work that would make the replacement valid. If the work could be shown, the intuition pump wouldn’t be very valuable, as one could just use the shown work for persuasion rather than the thought experiment with the disagreeing people. I strongly suspect that the reason the shown work is unavailable is because it does not exist.
Forget morality for one second. Doesn’t the meaning of the word “hat” differ from person to person?
It’s only sensible to say if/because context forestalls equivocation (or tries to, anyway). Retroactively removing the context by coming in the conversation with a different meaning of ought (even if the first meaning of “ought” was “objective values, as I think they are, as I think I want them to be, that are universally binding on all possible minds, and I would maintain under any coherent extrapolation of my values” where the first person is wrong about those facts and the second meaning of “ought” is the first person’s extrapolated volition) introduces equivocation. It’s really analogous to saying “No, I am not tall”.
Where the first person says “X would make me happy, I want to feel like doing X, and others will be better off according to balancing equation Y if I do X, and the word “ought” encompasses when those things coincide according to objective English, so I ought to do X”, and the second person says “X would make you happy, you want to feel like doing X, and others will not be better off according to balancing equation Z if you do X, and the word “ought” encompasses when those things coincide according to objective English, so you ought not do X”, they are talking past each other. Purported debates about the true meaning of “ought” reveal that everyone has their own balancing equation, and the average person thinks all others are morally obliged by objective morality to follow his or her equation. In truth, the terms “make happy” and want to feel like doing” are rolled up into the balancing equation, but in it (for Westerners) terms for self and others seem as if they are of different kind.
You’re confusing metaethics and first-order ethics. Ordinary moral debates aren’t about the meaning of “ought”. They’re about the first-order question of which actions have the property of being what we ought to do. People disagree about which actions have this property. They posit different systematic theories (or ‘balancing equations’ as you put it) as a hypothesis about which actions have the property. They aren’t stipulatively defining the meaning of ‘ought’, or else their claim that “You ought to follow the prescriptions of balancing equation Y” would be tautological, rather than a substantive claim as it is obviously meant to be.
I know that, which is why I said “Purported debates about the true meaning of ‘ought’” rather than “ordinary debates, which are about the true meaning of ‘ought’”.
Please be careful not to beg the question. People agree that there is such a property, but that is something about which they can be wrong.
Rather, they aren’t trying to stipulatively define the meaning of ‘ought’, or else their claim that “You ought to follow the prescriptions of balancing equation Y” would be tautological.
In fact, due to people’s poor self-insight, time limits, and the sometimes over-coarse granularity of language, they do not stipulate their actual balancing equation. Had they perfect insight and ability to represent their insights, it would be such a tautology. They would cease to speak like that had they the additional insight that for it to do the work it is called upon to do,”ought” is a word that needs grounding in the context of the real reasons for action of beings More generally, they are speaking an idiolect even regarding other definitions.
It’s meant to be such a claim, but it is in error because the speaker is confused about morality, and in a sense is not even wrong. They are claiming some actions have an objective moral valuation binding upon all intelligent beings, but they may as well claim the action has the property of being a square circle—or better yet, a perfect circle for pi is exactly 3, which is something I have witnessed a religious person claim is true.
~~~~~~~~~
I don’t understand either why you believe as you do or what good justification you might have for it.
I can see why one might want to make truth claims in which it falls out the folk have the least amount of confusion to be embarrassed about and are least wrong, and if one begins with the assumption that there are “moral facts” in the strongest sense, that’s a good start. However, that neither prevents one from having to say they are wrong about an enormous amount nor does it prevent one from having to claim that others, such as me, are wrong about moral facts. I can also see how it might be comforting were moral facts real in that sense and how the world might be better off if people believe in them, but obviously those things would not make them true.
At least you seem to understand the position you are arguing against and I am advocating. You repeatedly have tried to dissuade me from it by pointing out extensions of my position and counting on me to disclaim it from reductio ad absurdum. But it always seems to be by appealing to the same notion, that people simply cannot be wrong about some sorts of things they believe, which is a position I reject.
So, assuming I bite the bullet and don’t find it implausible that some people are fundamentally confused when they make moral claims, what other reasons might there be to agree with them? Why should I believe that intensionally defined “objective morality” has any extension at all?
I’m not arguing for moral realism here. I’m arguing against metaethical reductionism, which leaves open either realism OR error theory.
For all I’ve said, people may well be mistaken when they attribute normative properties to things. That’s fine. I’m just trying to clarify what it is that people are claiming when they make moral claims. This is conceptual analysis, not metaphysics. I’m pointing out that what you claim to be the meaning of ‘morality’ isn’t what people mean to be talking about when they engage in moral discourse. I’m not presupposing that ordinary people have any great insight into the nature of reality, but they surely do have some idea of what their own words mean. Your contrary linguistic hypothesis seems completely unfounded.
When I was young, I learned that the tooth fairy was really my mother all along.
What do you think of that?
(This isn’t meant to be insulting or anything similar.)
No, you learned that the tooth fairy doesn’t exist, and that your mother was instead responsible for the observable phenomena that you had previously attributed to the tooth fairy.
(It’s a good analogy though. I do think that claiming that morality exists “as a computation” is a lot like claiming that the tooth fairy really exists “as one’s mother”.)
Yes.
No.
Usually, the first thing to do when guessing about a random number from 1-100 is to split the possibilities in half by asking if it is more than 50 (or odd. Etc.)
The tooth fairy example gets a variety of responses, from people insisting it is just objectively wrong to say “the tooth fairy doesn’t exist” to those saying it is just objectively wrong to say the tooth fairy was really my mother. I happen to agree with you about what the best way is to describe what went on in this specific case. However, this is a standard blegg-rube situation that is unusual only in that it is not clear which way is best to describe the phenomenon to others.
There is a constellation of phenomena that correlate to each other—the fairy being female, being magic, having diaphanous wings, collecting things for money, those things being stored under pillows, those things being teeth. None of these is qualitatively essential to be a tooth fairy to most people than “having ten fingers” is essential to being human. If tomorrow we learn that magic is real, a female sprite collects teeth from under pillows, and does so on the back of a termite (and has size-changing technology/magic, why not?), most people would naively say “the tooth fairy does not fly, but burrows on the back of a termite”. That’s OK, but not great if the true nature of the situation is not recognized, and they fall into error if they think “tooth fairy” has a meaning divorced from flight. Likewise, those who say “there was never a ‘tooth fairy’, there is rather the ‘burrowing tooth fairy’” are right that there was never a thing exactly like the classic description, but this group makes an error if they demand the first stop calling the “burrowing tooth fairy” the “tooth fairy”.
There is more to say, an individual who makes up explanations ad hoc is not communicating, and the relative confluence of idiolects is valid because of the tinkerbell effect. that makes saying “No, you learned that the tooth fairy doesn’t exist” really peculiar, in the face of the many who hold the opposite position, likewise endorsed as language by common usage!
When it comes to “morality”, the constellation of things represented with a token labeled “second order morality” had several stars, significantly organized around a few (mutually contradictory, but all validly denoted by the word “morality”) sets of answers to these eight questions in the SEP on Moral Reasoning:
One characteristic of things often (and validly) called “morality” is that they include the following among other pieces to answer the above questions: “‘That which is good’ is an ephemeral property that is infused in things in a dualistic way, it can be separate from all concerns beings may have, “good” can be binding in a way that is tautological and pointless...etc.
Reductionism continues the tradition of “morality”, like other words, not meaning exactly what it did in the past, and it may be better to think of it as error theory in some cases. But all new theories claim the ones they supersede are in error, and there is nothing special about “morality minus magical deontology” or “moral reductionism” that one should consider them bad English by the usual standard.
After typing all of this, it strikes me that cars are a better example of no one component being essential. Cars without wheels will be “flying cars”, unless they get sufficient lift and air etc. such that as one goes down the line of models from flying jalopy to sports flying car, each successive flying car will be called “car” by fewer and fewer people as it more resembles a fighter jet or so...as perhaps our descendants will call nuclear-powered flying vehicles.
The important thing is that reductionism gets it right when describing reality, the problems with deontology etc., and explains why people think erroneously, and it then proceeds to tell one how to act such that first order moral claims have meaning. If despite all that, you want to focus on its erosion of folk theories to call it an error theory, that is understandable, but it will be uninteresting if you say to the multitude “It is wrong(RC) to call reductionism ‘moral’(RC),” since it is not wrong(everybody who recognizes not to misuse language by fixating on a connotation that often went along with a word when the contents described by that word are otherwise vindicated and under discussion, and we’re discussing language which means 1) constelations and 2) tinkerbell).
Seriously? I’ve never heard anyone insist that the tooth fairy really exists (in the form of their mother). It would seem most contrary to common usage (in my community, at least) to use ‘Tooth Fairy’ to denote “whoever replaced the tooth under my pillow with a coin”. The magical element is (in my experience) treated as essential to the term and not a mere “connotation”.
I’ve heard of the saying you mention, but I think you misunderstand people when you interpret it literally. My response was not intended as some “peculiar” declaration of mind-independent meaning facts, but rather as a straightforward interpretation of what people who utter such claims have in mind when they do so. (Ask them, “Do you mean that the tooth fairy exists?” and I expect the response, “No, silly, I just mean that my mother is responsible for the coin under my pillow.”)
So, to clarify: I don’t think that there are free-floating “meaning” facts out there independently of our linguistic dispositions. I just dispute whether your definitions adequately capture the things that most people really care about (i.e. treat as essential) when using the terms in question.
It’s no excuse to say that metaethical reductionism “gets reality right” when the whole dispute is instead over whether they have accommodated (or rather eliminated) some concept of which we have a pre-theoretic grasp. Compare the theological reductionist thesis that “God is love”. Love exists, therefore God exists, voila! If someone pointed out that this view is needlessly misleading since love is not what most people mean to be talking about when they speak of ‘God’ (and it would be more honest to just admit one’s atheism), it would be no response to give a lecture about constellations and tinkerbell.
What if metaethical reductionism is not meant (by some) to accommodate the pre-theoretic grasp of “morality” of most people, but just to accommodate the pre-theoretic grasp of “morality” of people like lessdazed? Could metaethical reductionism be considered a “respectable position” in that sense?
And separately, suppose the main reason I’m interested in metaethics is that I am trying to answer a question like “Should I terminally value the lives of random strangers?” and I’m not sure what that question means exactly or how I should go about answering it. In this case, is there a reason for me to care much about the pre-theoretic grasp of most people, as opposed to, say, people I think are most likely to be right about morality?
This is a good example, as people saying this are in some ways doing the opposite of what you advocate, but in other ways, they are doing the same thing.
I think people are motivated to say “God is love” out of a desire to avoid being logically compelled to view certain others as wrong (there is a twist, some passive aggressive non-believers say it to claim atheists are wrong when saying theists are wrong). The exact motivation isn’t important, but its presence would provide a countervailing force against the sheer silliness of their confusion (which is worse than yours in an important way) and explain how they could possibly make it.
The mistake being made is to pretend that there are inherently attached meanings to a word, as if those words simply had that meaning, regardless of context of using words in general, and that word in particular, when that context is totally adequate to explain meaning. When it is clear that contemporary theists and all their ancestors were in error, the hippie says “God is love” to pretend there is no disagreement, and that “God” meant love—instead of what was obviously meant, or often weirdly in addition to the contradictory thing that they admit was meant.
You do the opposite in that rather than seek ways to interpret others as agreeing or being right, even when there is obvious disagreement, you seek ways to interpret others as disagreeing or being wrong. You use the exact same mechanism of ignoring context and how language is used.
You avoid the worst error of the hippies, which is claiming that others mean both “God is love” and “God is an agent, etc.”
However, reductionists take the history of language and find words have many connotations, the meaning of “moral” and find it has a long history of meaning many different things, many of several things, and that the meaning people accept as associated to the term has to do with the quantity and quality of many aspects, none of which are intrinsic.
You have apparently decided that it’s magic or moral facts are the type of facts that can’t be explained in terms of how the world is is the element absolutely essential to “morality”, when (presumably, as so many random hundreds I’ve encountered accept the reductionist usage) thousands of others do not.
I doubt that were there a room full of hippies totally separated from theism that they would feel any need to obfuscate by saying “God is love”, which is why it is appropriate to barge in and correct the usage, as the usage is confusion designed to create the illusion of agreement.
You are attempting to create an illusion of disagreement about morality, where your disagreement is with how multitudes use “morality”—and that is not the sort of argument that it is possible to win because if that’s how we’re using it, that makes it right when we talk without confusion among each other, and reasonably use it with those whose linguistic sensibilities we don’t know explicitly. The reason why we’re doing so is because it most clearly communicates (out of all the words we could have used), and our stretching the meaning of “morality” to exclude magic is no more unusual than the stretching of the term “write” to include jabbing at keys with the fingers. We would continue to use it so in absence of non-reductionists, whereas hippies would revert to “love” in absence of theism.
Please understand, the disagreement is not about what happens but about what it means to literally be the tooth fairy.
That depends whether you are going to replace objective morality with subective morality or with error theory. It can be argued that subjectivism is an unstable position that amount to error theory.
This seems to me like begging the question. Can you expand on this?
Thinking more about this, it may have been better if Eliezer had not framed his meta-ethics sequence around “the meaning of right.”
If we play rationalist’s taboo with our moral terms and thus avoid moral terms altogether, what Eliezer seems to be arguing is that what we really care about is not (a) that whatever states of affairs our brains are wired to send reward signals in response to be realized, but (b) that we experience peace and love and harmony and discovery and so on.
His motivation for thinking this way is a thought experiment—which might become real in the relatively near future—about what would happen if a superintelligent machine could rewire our brains. If what we really care about is (a), then we shouldn’t object if the superintelligent machine rewires our brains to send reward signals only when we are sitting in a jar. But we would object to that scenario. Thus, what we care about seems not to be (a) but (b).
In a meta-ethicists terms, we could interpret Eliezer not as making an argument about the meaning of moral terms, but instead as making an argument that (b) is what gives us Reasons, not (a).
Now, all this meta-babble might not matter much. I’m pretty sure even if I was persuaded that the correct meta-ethical theory states that I should be okay with releasing a superintelligence that would rewire me to enjoy sitting in a jar, I would do whatever I could to prevent such a scenario and instead promote a superintelligence that would bring peace and joy and harmony and discovery and so on.
I thought being persuaded of a metaethical theory entails that whenever the theory tells you you should do X, you would feel compelled to do X.
This is a cool formulation. It’s interesting that there are other things that can happen to you not similar to “being persuaded of a metaethical theory” that entail that whenever you are told to do X you’re compelled to do X. (Voodoo or whatever.)
Only if motivational internalism is true. But motivational internalism is false.
What’s that?
Here, let me Google that for you.
I could get into how much I hate this kind of rejoinder if you bait me some more. I wasn’t asking you for the number of acres in a square mile. Let me just rephrase:
I hadn’t heard of motivational internalism before, could you expand your comment?
I don’t see what plausible reasoning process could lead you to infer this unlikely statement (about motivation, given how many detail would need to be just right for the statement to happen to be true).
Also, even if you forbid modifying definition of human brain, things that initiate high-reward signals in our brains (or that we actually classify as “harmony” or “love”) are very far from what we care about, just as whatever a calculator actually computes is not the same kind of consideration as the logically correct answer, even if you use a good calculator and aren’t allowed sabotage. There are many reasons (and contexts) for reward in human brain to not be treated as indicative of goodness of a situation.
I don’t understand your second paragraph. It sounds like you are agreeing to me, but your tone suggests you think you are disagreeing with me.
It was an explanation for why your thought experiment provides a bad motivation: we can just forbid modification of human brains to stop the thought experiment from getting through, but that would still leave a lot of problems, which shows that just this thought experiment is not sufficient motivation.
Sure, the superintelligence thought experiment is not the fully story.
One problem with the suggestion of writing a rule to not alter human brains comes in specifying how the machine is not allowed to alter human brains. I’m skeptical about our ability to specify that rule in a way that does not lead to disastrous consequences. After all, our brains are being modified all the time by the environment, by causes that are on a wide spectrum of ‘direct’ and ‘indirect.’
Other problems with adding such a rule are given here.
(I meant that subjective experience that evaluates situations should be specified using unaltered brains, not that brains shouldn’t be altered.)
You’ve got my curiosity. What does this mean? How would you realize that process in the real world?
Come on, this tiny detail isn’t worth the discussion. Classical solution to wireheading, asking the original and not the one under the influence, referring to you-at-certain-time and not just you-concept that resolves to something unpredicted at any given future time in any given possible world, rigid-designator-in-time.