There’s another big pile of gold, about 7,000 tonnes, in the New York Fed—that’s actually where a lot of foreign countries keep a large fraction of their gold supply. It’s open to tourists and you can walk in and look at the big stacks of gold bars. It does have fairly impressive security, but that security could plausibly be defeated by a reasonably competent wizard.
Jadagul
Karma: 200
I believe this is a misreading; Winky was there, but the Dark Mark was cast by Barry Crouch Jr. From the climax of Book 4, towards the end of Chapter 35:
I wanted to attack them for their disloyalty to my master. My father had left the tent; he had gone to free the Muggles. Winky was afraid to see me so angry. She used her own brand of magic to bind me to her. She pulled me from the tent, pulled me into the forest, away from the Death Eaters. I tried to hold her back. I wanted to return to the campsite. I wanted to show those Death Eaters what loyalty to the Dark Lord meant, and to punish them for their lack of it. I used the stolen wand to cast the Dark Mark into the sky.
The claim wasn’t that it happens too often to attribute to computation error, but that the types of differences seem unlikely to stem from computational errors.
You’re...very certain of what I understand. And of the implications of that understanding.
More generally, you’re correct that people don’t have a lot of direct access to their moral intuitions. But I don’t actually see any evidence for the proposition they should converge sufficiently other than a lot of handwaving about the fundamental psychological similarity of humankind, which is more-or-less true but probably not true enough. In contrast, I’ve seen lots of people with deeply, radically separated moral beliefs, enough so that it seems implausible that these all are attributable to computational error.
I’m not disputing that we share a lot of mental circuitry, or that we can basically understand each other. But we can understand without agreeing, and be similar without being the same.
As for the last bit—I don’t want to argue definitions either. It’s a stupid pastime. But to the extent Eliezer claims not to be a meta-ethical relativist he’s doing it purely through a definitional argument.
This comment may be a little scattered; I apologize. (In particular, much of this discussion is beside the point of my original claim that Eliezer really is a meta-ethical relativist, about which see my last paragraph).
I certainly don’t think we have to escalate to violence. But I do think there are subjects on which we might never come to agreement even given arbitrary time and self-improvement and processing power. Some of these are minor judgments; some are more important. But they’re very real.
In a number of places Eliezer commented that he’s not too worried about, say, two systems morality_1 and morality_2 that differ in the third decimal place. I think it’s actually really interesting when they differ in the third decimal place; it’s probably not important to the project of designing an AI but I don’t find that project terribly interesting so that doesn’t bother me.
But I’m also more willing to say to someone, “”We have nothing to argue about [on this subject], we are only different optimization processes.” With most of my friends I really do have to say this, as far as I can tell, on at least one subject.
However, I really truly don’t think this is as all-or-nothing as you or Eliezer seem to paint it. First, because while morality may be a compact algorithm relative to its output, it can still be pretty big, and disagreeing seriously about one component doesn’t mean you don’t agree about the other several hundred. (A big sticking point between me and my friends is that I think getting angry is in general deeply morally blameworthy, whereas many of them believe that failing to get angry at outrageous things is morally blameworthy; and as far as I can tell this is more or less irreducible in the specification for all of us). But I can still talk to these people and have rewarding conversations on other subjects.
Second, because I realize there are other means of persuasion than argument. You can’t argue someone into changing their terminal values, but you can often persuade them to do so through literature and emotional appeal, largely due to psychological unity. I claim that this is one of the important roles that story-telling plays: it focuses and unifies our moralities through more-or-less arational means. But this isn’t an argument per se and has no particular reason one would expect it to converge to a particular outcome—among other things, the result is highly contingent on what talented artists happen to believe. (See Rorty’s Contingency, Irony, and Solidarity for discussion of this).
Humans have a lot of psychological similarity. They also have some very interesting and deep psychological variation (see e.g. Haidt’s work on the five moral systems). And it’s actually useful to a lot of societies to have variation in moral systems—it’s really useful to have some altruistic punishers, but not really for everyone to be an altruistic punisher.
But really, this is beside the point of the original question, whether Eliezer is really a meta-ethical relativist, because the limit of this sequence which he claims converges isn’t what anyone else is talking about when they say “morality”. Because generally, “morality” is defined more or less to be a consideration that would/should be compelling to all sufficiently complex optimization processes. Eliezer clearly doesn’t believe any such thing exists. And he’s right.
Hm, that sounds plausible, especially your last paragraph. I think my problem is that I don’t see any reason to suspect that the expanded-enlightened-mature-unfolding of our present usages will converge in the way Eliezer wants to use as a definition. See for instance the “repugnant conclusion” debate; people like Peter Singer and Robin Hanson think the repugnant conclusion actually sounds pretty awesome, while Derek Parfit thinks it’s basically a reductio on aggregate utilitarianism as a philosophy and I’m pretty sure Eliezer agrees with him, and has more or less explicitly identified it as a failure mode of AI development. I doubt these are beliefs that really converge with more information and reflection.
Or in steven’s formulation, I suspect that relatively few agents actually have Ws in common; his definition presupposes that there’s a problem structure “implicitly defined by the machinery shared by X and Y which they both use to make desirability judgments”. I’m arguing that many agents have sufficiently different implicit problem structures that, for instance, by that definition Eliezer and Robin Hanson can’t really make “should” statements to each other.
I’m pretty sure Eliezer is actually wrong about whether he’s a meta-ethical relativist, mainly because he’s using words in a slightly different way from the way they use them. Or rather, he thinks that MER is using one specific word in a way that isn’t really kosher. (A statement which I think he’s basically correct about, but it’s a purely semantic quibble and so a stupid thing to argue about.)
Basically, Eliezer is arguing that when he says something is “good” that’s a factual claim with factual content. And he’s right; he means something specific-although-hard-to-compute by that sentence. And similarly, when I say something is “good” that’s another factual claim with factual content, whose truth is at least in theory computable.
But importantly, when Eliezer says something is “good” he doesn’t mean quite the same thing I mean when I say something is “good.” We actually speak slightly different languages in which the word “good” has slightly different meanings. Meta-Ethical Relativism, at least as summarized by wikipedia, describes this fact with the sentence “terms such as “good,” “bad,” “right” and “wrong” do not stand subject to universal truth conditions at all.” Eliezer doesn’t like that because in each speaker’s language, terms like “good” stand subject to universal truth conditions. But each speaker speaks a slightly different language where the truth conditions on the word represented by the string “good” stands subject to a slightly different set of universal truth conditions.
For an analogy: I apparently consistently define “blonde” differently from almost everyone I know. But it has an actual definition. When I call someone “blonde” I know what I mean, and people who know me well know what I mean. But it’s a different thing from what almost everyone else means when they say “blonde.” (I don’t know why I can’t fix this; I think my color perception is kinda screwed up). An MER guy would say that whether someone is “blonde” isn’t objectively true or false because what it means varies from speaker to speaker. Eliezer would say that “blonde” has a meaning in my language and a different meaning in my friends’ language, but in either language whether a person is “blonde” is in fact an objective fact.
And, you know, he’s right. But we’re not very good at discussing phenomena where two different people speak the same language except one or two words have different meanings; it’s actually a thing that’s hard to talk about. So in practice, “‘good’ doesn’t have an objective definition” conveys my meaning more accurately to the average listener than “‘good’ has one objective meaning in my language and a different objective meaning in your language.”
I was a grad student at Churchill, and we mostly ignored such things, but my girlfriend was an undergrad and felt compelled to educate me. I recall Johns being the rich kids, Peterhouse was the gay men (not sure if that’s for an actual reason or just the obvious pun), and a couple others that I can’t remember off the top of my head.
It’s mentioned, just not dwelled on. It’s mentioned once in passing in each of the first two books:
Sorceror’s Stone:
They piled so much homework on them that the Easter holidays weren’t nearly as much fun as the Christmas ones.
Chamber of Secrets:
The second years were given something new to think about during the Easter holidays.
And so on. It’s just that I don’t think anything interesting ever happens during them.
It occurred to me at some point that Fun Theory isn’t just the correct reply to Theodicy; it’s also a critical component of any religious theodicy program. And one of the few ways I could conceive of someone providing major evidence of God’s existence.
That is, I’m fairly confident that there is no god. But if I worked out a fairly complete version of Fun Theory, and it turned out that this really was the best of all possible worlds, I might have to change my mind.
I would agree with Karo, I think. I’m actually surprised by how accurate this list of predictions is; it’s not at 50% but I’m not sure why we would expect it to be with predictions this specific. (I’m not saying he was epistemically justified, just that he’s more accurate than I would have expected).
Following up on Eliezer’s point, it seems like the core of his claims are: 1) computers will become smaller and people will have access to them basically 24⁄7. If you remember that even my cell phone, which is a total piece of crap and cost $10, would look like a computer to someone from 1999, this seems fairly accurate. 2) Keyboards will cease to be the primary method of interaction with computers. We’re getting there—slowly—between improved voice recognition and the growth of tablet PCs. But we’re not there yet. I wouldn’t be surprised at all if this were true by 2020 (wouldn’t be surprised if it weren’t, either. I don’t know how far off we are from speech recognition good enough that people can assume it will work). 3) People will start using computers for things that in 1999 had to be done in hard copy. This is starting to happen, but we’re not there yet. Again, wouldn’t surprise me either way in 2020. 4) People will be able to use computers as their primary means of interaction with the world. Some basement-dwelling geeks like myself aside, not quite true. People like dealing with other people. I think this is the least likely to be true ten years from now.
Shane, the problem is that there are (for all practical purposes) infinitely many categories the Bayesian superintelligence could consider. They all “identify significant regularities in the environment” that “could potentially become useful.” The problem is that we as the programmers don’t know whether the category we’re conditioning the superintelligence to care about is the category we want it to care about; this is especially true with messily-defined categories like “good” or “happy.” What if we train it to do something that’s just like good except it values animal welfare far more (or less) than our conception of good says it ought to? How long would it take for us to notice? What if the relevant circumstance didn’t come up until after we’d released it?
This talk about metaethics is trying to justify building castles in the clouds by declaring the foundation to be supported by the roof. It doesn’t deal with the fundamental problem at all—it makes it worse.
Caledonian, I don’t want to speak for Eliezer. But my contention, at least, is that the fundamental problem is insoluble. I claim, not that this particular castle has a solid foundation, but that there exist no solid foundations, and that anywhere you think you’ve found solid earth there’s actually a cloud somewhere beneath it. The fact that you’re reacting so strongly makes me think you’re interpreting Eliezer as saying what I believe. Similarly,
Why should we care about a moral code that Eliezer has arbitrarily chosen to call right? What relevance does this have to anything?
There’s no particular reason we should care about a moral code Eliezer has chosen. You should care about the moral code you have arbitrarily chosen. I claim, and I think Eliezer would too, that there will be a certain amount of overlap because you’re both human (just as you both buy into Occam because you’re both human). But we couldn’t give, say, a pebblesorter any reason to care about Eliezer’s moral code.
Larry D’ana: Is anyone who does not believes in universally compelling arguments a relativist?
Is anyone who does not believe that morality is ontologically primitive a relativist?
Yeah, pretty much.
If there are no universally compelling arguments, then there’s no universally compelling moral code. Which means that whatever code compels you has to compel relative to who you are; thus it’s a relativist position.
Eliezer tries to get around this by saying that he has this code he can state (to some low degree of precision), and everyone can objectively agree on whether or not some action comports with this code. Or at least that perfect Bayesian superintelligences could all agree. (I’m not entirely sold on that, but we’ll stipulate). I claim, though, that this isn’t the way most people (including most of us) use the words ‘morality’ and ‘right’; I think that if you want your usage to comport with everyone else’s, you would have to say that the pebblesorters have ‘a’ moral code, and that this moral code is “Stack pebbles in heaps whose sizes are prime numbers.”
In other words, in general usage a moral code is a system of rules that compels an agent to action (and has a couple other properties I haven’t figured out how to describe without self-reference). A moral absolutist claims that there exists such a system of rules that is rightly binding and compelling to all X, where X is usually some set like “all human beings” or “all self-aware agents.” (Read e.g. Kant who claimed that the characteristic of a moral rule is that it is categorically binding on all rational minds). But Eliezer and I claim that there are no universally compelling arguments of any sort. Thus in particular there are no universally compelling injunctions to act, and thus no absolute moral code. Instead, the injunction to act that a particular agent finds compelling varies with the identity of the agent; thus ‘morality’ is relative to the agent. And thus I’m a moral relativist.
Now, it’s possible that you could get away with restricting X to “human beings”; if you then claimed that humans had enough in common that the same moral code was compelling to all of them, you could plausibly reclaim moral objectivism. But I think that claim is clearly false; Eliezer seems to have rejected it (or at least refused to defend it) as well. So we don’t get even that degree of objectivity; the details of each person’s moral code depend on that person, and thus we have a relative standard. This is what has Caledonian’s knickers in such a twist.
Kenny: exactly. That’s why we’re morally relative.
Eliezer: Good post, as always, I’ll repeat that I think you’re closer to me in moral philosophy than anyone else I’ve talked to, with the probable exception of Richard Rorty, from whom I got many of my current views. (You might want to read Contingency, Irony, Solidarity; it’s short, and it talks about a lot of the stuff you deal with here). That said, I disagree with you in two places. Reading your stuff and the other comments has helped me refine what I think; I’ll try to state it here as clearly as possible.
1) I think that, as most people use the words, you’re a moral relativist. I understand why you think you’re not. But the way most people use the word ‘morality,’ it would only apply to an argument that would persuade the ideal philosopher of perfect emptiness. You don’t believe any such arguments exist; neither do I. Thus neither of us think that morality as it’s commonly understood is a real phenomenon. Think of the priest in War of the Worlds who tried to talk to the aliens, explaining that since we’re both rational beings/children of God, we can persuade them not to kill us because it’s wrong. You say (as I understand you) that they would agree that it’s wrong, and just not care, because wrong isn’t necessarily something they care about. I have no problem with any claim you’ve made (well, that I’ve made on your behalf) here; but at this point the way you’re using the word ‘moral’ isn’t a way most people would use it. So you should use some other term altogether.
2) I like to maintain a clearer focus on the fact that, if you care about what’s right, I care about what’s right_1, which is very similar to but not the same as what’s right. Mainly because it helps me to remember there are some things I’m just not going to convince other people of (e.g. I don’t think I could convince the Pope that God doesn’t exist. There’s no fact pattern that’s wholly inconsistent with the property god_exists, and the Pope has that buried deep enough in its priors that I don’t think it’s possible to root it out). But (as of reading your comment on yesterday’s post) I don’t think we disagree on the substance, just on the emphasis.
Thanks for an engaging series of posts; as I said, I think you’re the closest or second-closest I’ve ever come across to someone sharing my meta-ethics.
Ah, thanks Eliezer, that comment explains a lot. I think I mostly agree with you, then. I suspect (on little evidence) that each one of us would, extrapolated, wind up at his own attractor (or at least at a sparsely populated one). But I have no real evidence for this, and I can’t imagine off the top of my head how I would find it (nor how I would find contradictory evidence), and since I’m not trying to build fAI I don’t need to care. But what you’ve just sketched out is basically the reason I think we can still have coherent moral arguments; our attractors have enough in common that many arguments I would find morally compelling, you would also find morally compelling (as in, most of us have different values but we (almost) all agree that the random slaughter of innocent three-year-olds is bad). Thanks for clearing that up.
Especially given that exposure to different fact patterns could push you in different directions. E.g. suppose right now I try to do what is right_1 (subscripts on everything to avoid appearance of claim to universality). Now, suppose that if I experience fact pattern facts_1 I conclude that it is right_1 to modify my ‘moral theory’ to right_2. but if I experience fact pattern facts_2 I conclude that it is right_1 to modify to right_3.
Now, that’s all well and good. Eliezer would have no problem with that, as long as the diagram commutes: that is, if it’s true that ( if I’ve experienced facts_1 and moved to right_2, and then I experience facts_2, I will move to right_4), it must also be true that ( if I’ve experienced facts_2 and moved to right_3, and then experience facts_1, I will move to right_4).
I suppose that at least in some cases this is true, but I see no reason why in all cases it ought to be. Especially if you allow human cognitive biases to influence the proceedings; but even if you don’t (and I’m not sure how you avoid it), I don’t see any argument why all such diagrams should commute. (this doesn’t mean they don’t, of course. I invite Eliezer to provide such an argument).
I still hold that Eliezer’s account of morality is correct, except his claim that all humans would reflectively arrive at the same morality. I think foundations and priors are different enough that, functionally, each person has his own morality.
But Mario, why not? In J-morality it’s wrong to hurt people, both because I have empathy towards people and so I like them, and because people tend to create net positive externalities. But that’s a value judgment. I can’t come up with any argument that would convince a sociopath that he “oughtn’t” kill people when he can get away with it. Even in theory.
There was nothing wrong with Raskolnikov’s moral theory. He just didn’t realize that he wasn’t a Napoleon.
Eliezer, I think you come closer to sharing my understanding of morality than anyone else I’ve ever met. Places where I disagree with you:
First, as a purely communicative matter, I think you’d be clearer if you replaced all instances of “right” and “good” with “E-right” and “E-good.”
Second, as I commented a couple threads back, I think you grossly overestimate the psychological unity of humankind. Thus I think that, say, E-right is not at all the same as J-right (although they’re much more similar than either is to p-right). The fact that our optimization processes are close enough in many cases that we can share conclusions and even arguments doesn’t mean that they’re the same optimization process, or that we won’t disagree wildly in some cases.
Simple example: I don’t care about the well-being of animals. There’s no comparison in there, and there’s no factual claim. I just don’t care. When I read the famous ethics paper about “would it be okay to torture puppies to death to get a rare flavor compound,” my response was something along the lines of, “dude, they’re puppies. Who cares if they’re tortured?” I think anyone who enjoys torturing for the sake of torturing is probably mentally unbalanced and extremely unvirtuous. But I don’t care about the pain in the puppy at all. And the only way you could make me care is if you showed that causing puppies pain came back to affect human well-being somehow.
Third, I think you are a moral relativist, at least as that claim is generally understood. Moral absolutists typically claim that there is some morality demonstrably binding upon all conscious agents. You call this an “attempt to persuade an ideal philosopher of perfect emptiness” and claim that it’s a hopeless and fundamentally stupid task. Thus you don’t believe what moral absolutists believe; instead, you believe different beings embody different optimization processes (which is the name you give to what most people refer to as morality, at least in conscious beings). You’re a moral relativist. Which is good, because it means you’re right.
Excuse me. It means you’re J-right.
Caledonian and Tim Tyler: there are lots of coherent defenses of Christianity. It’s just that many of them rest on statements like, “if Occam’s Razor comes into conflict with Revealed Truth, we must privilege the latter over the former.” This isn’t incoherent; it’s just wrong. At least from our perspective. Which is the point I’ve been trying to make. They’d say the same thing about us.
Roko: I sent you an email.
Canon is fairly clear that Hogwarts is the only game in Britain. It also leads to glaring inconsistencies in scale which you just pointed out. (Rowling originally said that Hogwarts had about 700 students, and then fans started pointing out that that was wildly inconsistent with the school as she described it. And even that’s too small to make things really work).
But the evidence, from HP7 (page 210 of my first-run American hardback copy):
Lupin is talking about Voldemort’s takeover of Wizarding society, to Harry and the others.
“Most wizards” in Britain were educated at Hogwarts, and the exceptions were homeschooled or sent abroad. It’s really hard to read that to imply that there’s another British wizarding school anywhere.