Well, nice to know we’re planning our global thermonuclear wars decades before there’s any sign we’ll need a global thermonuclear war for any good reason.
Goddamnit, do you people just like plotting wars!?
You’re equivocating on the word “need”. When one refers to needing most things, it means we’re better off with them than with not having them. But for global thermonuclear war, the comparison is not to having no war; the comparison is to having a war where other parties are the ones with all the nukes.
Furthermore, describing many actions in terms of “need” is misleading. “Needing” something normally implies a naive model where if you want X to happen, you are willing to do X and vice versa. Look up everything that has been written here about precommitting; nuclear war is a case of precommitting and precommitting to something can actually reduce its likelihood.
You’re equivocating on the word “need”. When one refers to needing most things, it means we’re better off with them than with not having them. But for global thermonuclear war, the comparison is not to having no war; the comparison is to having a war where other parties are the ones with all the nukes.
No, we’re not talking about that kind of war. We’re not talking about a balance of power that can be maintained through anti-proliferation laws (though I certainly support international agreements to not build AI and contribute to a shared, international FAI project!). If we get to the point of an American FAI versus a Chinese FAI, the two AIs will negotiate a rational compromise to best suit the American and Chinese CEVs (which won’t even be that different compared to, say, Clippy).
Whereas if we get one UFAI that manages to go FOOM, it doesn’t fucking matter who built it: we’re all dead.
So the issue is not, “You don’t build UFAI and I won’t build UFAI.” The issue is simply: don’t build UFAI, ever, at all. All humans have rational reason to buy this proposition.
There are actually two better options here than preemptively plotting an existential-risk-grade war. They are not dichotomous and I personally support employing both.
Plot an international treaty to limit the creation of FOOM-able AIs outside a strict framework of cooperative FAI development that involves a broad scientific community and limits the resources needed for rogue states or organizations to develop UFAI. This favors the singleton approach advocated by Nick Bostrom and Eliezer Yudkowsky, and also avoids thermonuclear war. An Iraq-style conventional war of regime change is already a severe enough threat to bend most nations’ interests in favor of either cooperative FAI development or just not developing AI.
For the case of a restricted-domain FAI being created, encourage global economic cooperation and cultural interaction, to ensure that whether the first FAI is Chinese or American, it will infer values over humans of a more global rather than parochial culture and orientation (though I had thought Eliezer’s cognitivist approach to human ethics was meant to be difficult to corrupt using mere cultural brainwashing).
That leaves the following military options: in case of a regime showing signs of going rogue and developing their own AI, utilize conventional warfare (which in an increasingly economically interconnected world is already extremely painful for anyone except North Korea or the very poor, neither of which are good at building AIs). In case of an actual UFAI appearing and beginning a process of paper-clipping the world within a timespan that we can see it coming before it kills us: consider annihilating the planet.
However, strangely enough, none of these options suit the cultural fetish around here for sitting around in hooded cloaks plotting the doom of others in secret and feeling ever-so-”rational” about ourselves for being willing to engage in deception, secrecy, and murder for the Greater Good. So I predict people here won’t actually want to take those options, because the terminal goal at work is Be Part of the Conspiracy rather than Ensure the First Superintelligence is Friendly.
none of these options suit the cultural fetish around here for sitting around in hooded cloaks plotting the doom of others in secret and feeling ever-so-”rational” about ourselves for being willing to engage in deception, secrecy, and murder for the Greater Good.
Frankly, when someone writes a post recommending global thermonuclear war as a possible option, that’s my line. My suggested courses of action are noticeably less melodramatic and noticeably closer to the plain, boring field of WW3-prevention.
But I gave you the upvote anyway for calling out my davkanik tendencies.
Frankly, when someone writes a post recommending global thermonuclear war as a possible option, that’s my line.
I’m genuinely confused. There’s an analogy to a nuclear arms race running through the OP, but as best I can tell it’s mostly linking AI development controls to Cold War-era arms control efforts—which seems reasonable, if inexact. Certainly it’s not advocating tossing nukes around.
Can you point me to exactly what you’re responding to?
Ah, I seem to be referring to James’ excerpt from his book rather than the OP:
A friendly AI would allow trillions and trillions of people to eventually live their lives, and mankind and our descendents could survive to the end of the universe in utopia. In contrast, an unfriendly AI would destroy us. I have decided to make the survival of mankind my overwhelming priority. Consequently, since a thermonuclear war would non-trivially increase the chance of mankind’s survival, I believe that it’s my moral duty to initiate war, even though my war will kill over a billion human beings.
Oh, that makes more sense. I’d assumed, since this thread was rooted under the OP, that you were responding to that.
After reading James’s post, though, I don’t think it’s meant to be treated as comprehensive, much less prescriptive. He seems to be giving some (fictional) outlines of outcomes that could arise in the absence of early and aggressive cooperation on AI development; the stakes at that point are high, so the consequences are rather precipitous, but this is still something to avoid rather than something to pursue. Reading between the lines, in fact, I’d say the policy implications he’s gesturing towards are much the same as those you’ve been talking about upthread.
On the other hand, it’s very early to be hashing out scenarios like this, and doing so doesn’t say anything particularly good about us from a PR perspective. It’s hard enough getting people to take AI seriously as a risk, full stop; we don’t need to exacerbate that with wild apocalyptic fantasies just yet.
It’s hard enough getting people to take AI seriously as a risk, full stop
This bears investigating. I mean, come on, the popular view of AI among the masses is that All AI Is A Crapshoot, that every single time it will end in the Robot Wars. So how on Earth can it be difficult to convince people that UFAI is an issue?
I mean, hell, if I wanted to scare someone, I’d just point out that no currently-known model of AGI includes a way to explicitly specify goals desirable to humans. That oughtta scare folks.
I’ve talked to a number of folks who conclude that AIs will be superintelligent and therefore will naturally derive and follow the true morality (you know, the same one we do), and dismiss all that Robot Wars stuff as television crap (not unreasonably, as far as it goes).
I’ve never had that conversation with explicitly religious people, and moral realism at the “some things are just wrong and any sufficiently intelligent system will know it” level is hardly unheard of among atheists.
moral realism at the “some things are just wrong and any sufficiently intelligent system will know it” level is hardly unheard of among atheists.
Really? I mean, sorry for blathering, but I find this extremely surprising. I always considered it a simple fact that if you don’t have some kind of religious/faith-based metaphysics operating, you can’t be a moral realist. What experiment could you possibly perform to test moral-realist hypotheses, particularly when dealing with nonhumans? It simply doesn’t make any sense.
Disagreed, depending on your definition of “morality”. A sufficiently totalitarian God can easily not only decide what is moral but force us to find the proper morality morally compelling.
(There is at least one religion that actually believes something along these lines, though I don’t follow it.)
Ok, that definition is not nonsense. But in that case, it could happen without God too. Maybe the universe’s laws cause people to converge on some morality, either due to the logic of evolutionary cooperation or another principle. It could even be an extra feature of physics that forces this convergence.
Perhaps Eli and you are talking past each other a bit. A certain kind of god would be strong evidence for moral realism, but moral realism wouldn’t be strong evidence for a god of any kind.
You talk as though religion were something that appeared in people’s minds fully formed and without causes, and that the logical fallacies associated with it were then caused by religion.
Well no, not really. The meta-ethics sequence takes a cognitivist position: there is some cognitive algorithm called ethics, which actual people implement imperfectly but which you could somehow generalize to obtain a “perfect” reification.
That’s not moral realism (“morality is a part of the universe itself, external to human beings”), that’s objective moral-cognitivism (“morality is a measurable part of us but has no other grounding in external reality”).
Well if you knew what the goals were and could prove that such goals appeal to all intelligent, rational beings, including but not limited to humans, UFAI, Great Cthulhu, and business corporations...
I don’t need to do that. We are used to the idea that some people don’t find morality appealing, and we have mechanisms such as social disapproval and prisons to get the recalcitrant to play along.
That depends: what are you talking about? I seem to recall you defined the term as something that Eliezer might agree with. If you’ve risen to the level of clear disagreement, I haven’t seen it.
A good refinement of the question is how you think AI could go wrong (that being Eliezer’s field) if we reject whatever you’re asking about.
You would have the exact failure mode you are already envisaging...clippies and so on. OMC is a way .AI would not go wrong. MIRI needs to argue it is unfeasible or unlikely to show that uFAI is likely.
Really? I mean, sorry for blathering, but I find this extremely surprising. I always considered it a simple fact that if you don’t have some kind of religious/faith-based metaphysics operating, you can’t be a moral realist. What experiment could you possibly perform
That would be epistemology...
to test moral-realist hypotheses, particularly when dealing with nonhumans? It simply doesn’t make any sense.
There are rationally acceptable subjects that don’t use empiricism, such as maths, and there are subjects such as economics which have a mixed epistemology.
However, if this epistemological-sounding complaint is actually about metaphysics, ie “what experiment could you perform to detect a non-natural moral property”, the answer is that moral realists have to suppose the existence of special psychological faculty.
Pedantic complaint about language: moral realism simply says that moral claims do state facts, and at least some of them are true. It takes further assumptions (“internalism”) to claim that these moral facts are universally compelling in the sense of moving any intelligent being to action. (I personally believe the latter assumption to be nonsense, hence AGI is a really bad idea.)
Granted, I don’t know of any nice precise term for that position that all intelligent beings must necessarily do the right thing, possibly because it’s so ridiculous no philosopher would profess it publicly in such words. On the other hand, motivational internalism would seem to be very intuitive, judging by the pervasiveness of the view that AI doesn’t pose any risk.
So, moral motivational internalism. Then I agree that we tend to reject it. For example, here. You can make it work by having “this motivates the person considering it” be incorporated into the definition of “right”, but that results in a relativist definition, and I don’t see any need for it anyway.
No, the idea of motivational internalism is that you can’t judge something as right or wrong without being motivated to pursue or avoid it. Like if the word “right” was short for “this thing matches my terminal values”.
The alternative is externalism, where “right” means {X, Y, Z} and we (some/most/all humans) are motivated to pursue it just because we like {X, Y, Z}.
Does “Intrinsic Motivation” in this context entail that all intelligent beings must necessarily do the right thing?
If so, then I agree that we tend to reject it. As for “without argument”… do you mean you’ve read the local discussions of the topic and find them unconvincing? Or do you mean you believe it hasn’t been discussed at all?
If not, then I don’t know what you’re saying.
If you prefer to continue expressing yourself in gnomic utterances, that’s of course your choice, but I find it an unhelpful way to communicate and will tap out here if so.
Eh, maybe? I’ve seen “convergence thesis” thrown about on LW, but it’s hardly established terminology. Not sure it would be fair to use a phrase so easily confused with Bostrom’s much more reasonable Instrumental Convergence Thesis either. (Also, it has nothing to do with CEV so I don’t see the point of that link.)
Are these religious people? I mean, come on, where do you get moral realism if not from some kind of moral metaphysics?
From abstract reason or psychological facts, or physical facts, or a mixture.
There is a subject called economics. It tells you how to achieve certain goals, such as maximising GDP. It doesn’t do that by corresponding to a metaphysical Economics Object, it does that with a mixture of theoretical reasoning and examination of evidence.
There is a subject called ethics. It tells you how to achieve certain goals, such as maximising happiness....
There is a subject called ethics. It tells you how to achieve certain goals, such as maximising happiness....
Well there’s the problem: ethics does not automatically start out with a happiness-utilitarian goal. Lots of extent ethical systems use other terminal goals. For instance...
Of course economics doesn’t have the well-established laws of physical science: it wouldn’t be much of an analogy for ethics if it did.But having an epistemology that doens’t work very well is not the same as having an epistemology that requires non-natural entities.
The main problem with economics is not its descriptive, but its predictive power. Too many of economics’ calculations need to suppose that everyone will behave rationally, which regular people can’t be trusted to do. Same problem with politics.
So how on Earth can it be difficult to convince people that UFAI is an issue?
Well, there’s a couple prongs to that. For one thing, it’s tagged as fiction in most people’s minds, as might be suggested by the fact that it’s easily described in trope. That’s bad enough by itself.
Probably more importantly, though, there’s a ferocious tendency to anthropomorphize this sort of thing, and you can’t really grok UFAI without burning a good bit of that tendency out of your head. Sure, we ourselves aren’t capital-F Friendly, but we’re a far cry yet from a paperclip maximizer or even most of the subtler failures of machine ethics; a jealous or capricious machine god is bad, but we’re talking Screwtape here, not Azathoth. HAL and Agent Smith are the villains of their stories, but they’re human in most of the ways that count.
You may also notice that we tend to win fictional robot wars.
Also, note that the tropes tend to work against people who say “we have a systematic proof that our design of AI will be Friendly”. In fact, in general the only way a fictional AI will turn out ‘friendly’ is if it is created entirely by accident—ANY fictional attempt to intentionally create a Friendly AI will result in an abomination, usually through some kind of “dick Genie” interpretation of its Friendliness rules.
Yeah. I think I’d consider that a form of backdoor anthropomorphization by way of vitalism, though. Since we tend to think of physically nonhuman intelligences as cognitively human, and since we tend to think of human ethics and cognition as something sacred and ineffable, fictional attempts to eff them tend to be written as crude morality plays.
Intelligence arising organically from a telephone exchange or an educational game or something doesn’t trigger the same taboos.
Frankly, when someone writes a post recommending global thermonuclear war as a possible option, that’s my line. My suggested courses of action are noticeably less melodramatic and noticeably closer to the plain, boring field of WW3-prevention.
The currently fashionable descriptor is “metacontrarianism”—you might get better responses if you phrase your objection in that way.
when someone writes a post recommending global thermonuclear war as a possible option
Looks like you (emphasis mine):
In case of an actual UFAI appearing and beginning a process of paper-clipping the world within a timespan that we can see it coming before it kills us: consider annihilating the planet
and
my davkanik tendencies
You can be a contrarian with less drama perfectly well :-)
I would note that “we are all in the process of dying horribly” is actually a pretty dramatic situation. At the moment, actually, I’m not banking on ever seeing it: I think actual AI creation requires such expertise and has such extreme feasibility barriers that successfully building a functioning software-embodied optimization process tends to require such group efforts that someone thinks hard about what the goal system is.
I would note that “we are all in the process of dying horribly” is actually a pretty dramatic situation.
Given that “we are all in the process of dying” is true for all living beings for as long as living beings existed, I don’t see anything dramatic in here. As to “horribly”, what is special about today’s “horror” compared to, say, a hundred years ago?
I hadn’t meant today. I had meant in the case of a UFAI getting loose. That’s one of those rare situations where you should consider yourself assuredly dead already and start considering how you’re going to kill the damn UFAI, whatever that costs you.
Whereas in the present day, I would not employ “nuke it from orbit; only way to be sure” solutions to, well, anything.
Thanks, Eli. You make some good points amidst the storm. :)
I think the scenario James elaborated was meant to be a fictional portrayal of a bad outcome that we should seek to avoid. That it was pasted without context may have given the impression that he actually supported such a strategy.
I mostly agree with your bullet points. Working toward cooperation and global unification, especially before things get ugly, is what I was suggesting in the opening post.
Even if uFAI would destroy its creators, people still have incentive to skimp on safety measures in an arms-race situation because they’re trading off some increased chance of winning against some increased chance of killing everyone. If winning the race is better than letting someone else win, then you’re willing to tolerate some increased risk of killing everyone. This is why I suggested promoting internationalist perspective as one way to improve the situation—because then individual countries would care less about winning the race.
BTW, it’s not clear that Clippy would kill us all. Like in any other struggle for power, a newly created Clippy might compromise with humans by keeping them alive and giving them some of what they want. This is especially likely if Clippy is risk averse.
Interesting. So there are backup safety strategies. That’s quite comforting to know, actually.
I think the scenario James elaborated was meant to be a fictional portrayal of a bad outcome that we should seek to avoid. That it was pasted without context may have given the impression that he actually supported such a strategy.
Oh thank God. I’d like to apologize for my behavior, but to be honest this community is oftentimes over my Poe’s Law Line where I can no longer actually tell if someone is acting out a fictional parody of a certain idea or actually believes in that idea.
Next time I guess I’ll just assign much more probability to the “this person is portraying a fictional hypothetical” notion.
If winning the race is better than letting someone else win, then you’re willing to tolerate some increased risk of killing everyone.
Sorry, could you explain? I’m not seeing it. That is, I’m not seeing how increasing the probability that your victory equates with your own suicide is better than letting someone else just kill you. You’re dead either way.
That is, I’m not seeing how increasing the probability that your victory equates with your own suicide is better than letting someone else just kill you. You’re dead either way.
Say that value(you win) = +4, value(others win) = +2, value(all die) = 0. If you skimp on safety measures for yourself, you can increase your probability of winning relative to others, and this is worth some increased chance of killing everyone. Let me know if you want further clarification. :) The final endpoint of this process will be a Nash equilibrium, as discussed in “Racing to the Precipice,” but what I described could be one step toward reaching that equilibrium.
Well, nice to know we’re planning our global thermonuclear wars decades before there’s any sign we’ll need a global thermonuclear war for any good reason.
Goddamnit, do you people just like plotting wars!?
You’re equivocating on the word “need”. When one refers to needing most things, it means we’re better off with them than with not having them. But for global thermonuclear war, the comparison is not to having no war; the comparison is to having a war where other parties are the ones with all the nukes.
Furthermore, describing many actions in terms of “need” is misleading. “Needing” something normally implies a naive model where if you want X to happen, you are willing to do X and vice versa. Look up everything that has been written here about precommitting; nuclear war is a case of precommitting and precommitting to something can actually reduce its likelihood.
No, we’re not talking about that kind of war. We’re not talking about a balance of power that can be maintained through anti-proliferation laws (though I certainly support international agreements to not build AI and contribute to a shared, international FAI project!). If we get to the point of an American FAI versus a Chinese FAI, the two AIs will negotiate a rational compromise to best suit the American and Chinese CEVs (which won’t even be that different compared to, say, Clippy).
Whereas if we get one UFAI that manages to go FOOM, it doesn’t fucking matter who built it: we’re all dead.
So the issue is not, “You don’t build UFAI and I won’t build UFAI.” The issue is simply: don’t build UFAI, ever, at all. All humans have rational reason to buy this proposition.
There are actually two better options here than preemptively plotting an existential-risk-grade war. They are not dichotomous and I personally support employing both.
Plot an international treaty to limit the creation of FOOM-able AIs outside a strict framework of cooperative FAI development that involves a broad scientific community and limits the resources needed for rogue states or organizations to develop UFAI. This favors the singleton approach advocated by Nick Bostrom and Eliezer Yudkowsky, and also avoids thermonuclear war. An Iraq-style conventional war of regime change is already a severe enough threat to bend most nations’ interests in favor of either cooperative FAI development or just not developing AI.
For the case of a restricted-domain FAI being created, encourage global economic cooperation and cultural interaction, to ensure that whether the first FAI is Chinese or American, it will infer values over humans of a more global rather than parochial culture and orientation (though I had thought Eliezer’s cognitivist approach to human ethics was meant to be difficult to corrupt using mere cultural brainwashing).
That leaves the following military options: in case of a regime showing signs of going rogue and developing their own AI, utilize conventional warfare (which in an increasingly economically interconnected world is already extremely painful for anyone except North Korea or the very poor, neither of which are good at building AIs). In case of an actual UFAI appearing and beginning a process of paper-clipping the world within a timespan that we can see it coming before it kills us: consider annihilating the planet.
However, strangely enough, none of these options suit the cultural fetish around here for sitting around in hooded cloaks plotting the doom of others in secret and feeling ever-so-”rational” about ourselves for being willing to engage in deception, secrecy, and murder for the Greater Good. So I predict people here won’t actually want to take those options, because the terminal goal at work is Be Part of the Conspiracy rather than Ensure the First Superintelligence is Friendly.
Oh, how… rebel of you.
May I recommend less drama?
Frankly, when someone writes a post recommending global thermonuclear war as a possible option, that’s my line. My suggested courses of action are noticeably less melodramatic and noticeably closer to the plain, boring field of WW3-prevention.
But I gave you the upvote anyway for calling out my davkanik tendencies.
I’m genuinely confused. There’s an analogy to a nuclear arms race running through the OP, but as best I can tell it’s mostly linking AI development controls to Cold War-era arms control efforts—which seems reasonable, if inexact. Certainly it’s not advocating tossing nukes around.
Can you point me to exactly what you’re responding to?
Ah, I seem to be referring to James’ excerpt from his book rather than the OP:
Oh, that makes more sense. I’d assumed, since this thread was rooted under the OP, that you were responding to that.
After reading James’s post, though, I don’t think it’s meant to be treated as comprehensive, much less prescriptive. He seems to be giving some (fictional) outlines of outcomes that could arise in the absence of early and aggressive cooperation on AI development; the stakes at that point are high, so the consequences are rather precipitous, but this is still something to avoid rather than something to pursue. Reading between the lines, in fact, I’d say the policy implications he’s gesturing towards are much the same as those you’ve been talking about upthread.
On the other hand, it’s very early to be hashing out scenarios like this, and doing so doesn’t say anything particularly good about us from a PR perspective. It’s hard enough getting people to take AI seriously as a risk, full stop; we don’t need to exacerbate that with wild apocalyptic fantasies just yet.
This bears investigating. I mean, come on, the popular view of AI among the masses is that All AI Is A Crapshoot, that every single time it will end in the Robot Wars. So how on Earth can it be difficult to convince people that UFAI is an issue?
I mean, hell, if I wanted to scare someone, I’d just point out that no currently-known model of AGI includes a way to explicitly specify goals desirable to humans. That oughtta scare folks.
I’ve talked to a number of folks who conclude that AIs will be superintelligent and therefore will naturally derive and follow the true morality (you know, the same one we do), and dismiss all that Robot Wars stuff as television crap (not unreasonably, as far as it goes).
Which one’s that, eh ;-)?
Are these religious people? I mean, come on, where do you get moral realism if not from some kind of moral metaphysics?
Certainly it’s not unreasonable. One UFAI versus humans with no FAI to fight back, I wouldn’t call anything so one-sided a war.
(And I’m sooo not making the Dalek reference that I really want to. Someone else should do it.)
I’ve never had that conversation with explicitly religious people, and moral realism at the “some things are just wrong and any sufficiently intelligent system will know it” level is hardly unheard of among atheists.
Really? I mean, sorry for blathering, but I find this extremely surprising. I always considered it a simple fact that if you don’t have some kind of religious/faith-based metaphysics operating, you can’t be a moral realist. What experiment could you possibly perform to test moral-realist hypotheses, particularly when dealing with nonhumans? It simply doesn’t make any sense.
Oh well.
Moral realism makes no more sense with religion. As CS Lewis said: “Nonsense does not cease to be nonsense when we put the words ‘God can’ before it.”
Disagreed, depending on your definition of “morality”. A sufficiently totalitarian God can easily not only decide what is moral but force us to find the proper morality morally compelling.
(There is at least one religion that actually believes something along these lines, though I don’t follow it.)
Ok, that definition is not nonsense. But in that case, it could happen without God too. Maybe the universe’s laws cause people to converge on some morality, either due to the logic of evolutionary cooperation or another principle. It could even be an extra feature of physics that forces this convergence.
Perhaps Eli and you are talking past each other a bit. A certain kind of god would be strong evidence for moral realism, but moral realism wouldn’t be strong evidence for a god of any kind.
Well sure, but if you’re claiming physics enforces a moral order, you’ve reinvented non-theistic religion.
Why? Beliefs that make no sense are very common. Atheists are no exception.
Actually, if anything, I’d call it the reverse. Religious people know where we’re making unevidenced assumptions.
You talk as though religion were something that appeared in people’s minds fully formed and without causes, and that the logical fallacies associated with it were then caused by religion.
Hmm. Fair point. “We imagine the universe as we are.”
Might I suggest you take a look at the metaethics sequence? This position is explained very well.
Well no, not really. The meta-ethics sequence takes a cognitivist position: there is some cognitive algorithm called ethics, which actual people implement imperfectly but which you could somehow generalize to obtain a “perfect” reification.
That’s not moral realism (“morality is a part of the universe itself, external to human beings”), that’s objective moral-cognitivism (“morality is a measurable part of us but has no other grounding in external reality”).
Can you rule out a form objective moral-cognitivism that applies to any sufficiently rational and intelligent being?
Unless morality consists of game theory, I can rule out any objective moral cognitivism that applies to any intelligent and/or rational being.
Why shouldn’t it consist of optimal rules for achieving certain goals?
Well if you knew what the goals were and could prove that such goals appeal to all intelligent, rational beings, including but not limited to humans, UFAI, Great Cthulhu, and business corporations...
I don’t need to do that. We are used to the idea that some people don’t find morality appealing, and we have mechanisms such as social disapproval and prisons to get the recalcitrant to play along.
That depends: what are you talking about? I seem to recall you defined the term as something that Eliezer might agree with. If you’ve risen to the level of clear disagreement, I haven’t seen it.
A good refinement of the question is how you think AI could go wrong (that being Eliezer’s field) if we reject whatever you’re asking about.
You would have the exact failure mode you are already envisaging...clippies and so on. OMC is a way .AI would not go wrong. MIRI needs to argue it is unfeasible or unlikely to show that uFAI is likely.
Which position? The metaethics sequence isn’t clearly re4alist, or anything else.
That would be epistemology...
There are rationally acceptable subjects that don’t use empiricism, such as maths, and there are subjects such as economics which have a mixed epistemology.
However, if this epistemological-sounding complaint is actually about metaphysics, ie “what experiment could you perform to detect a non-natural moral property”, the answer is that moral realists have to suppose the existence of special psychological faculty.
You seem to be confusing atheism with positivism. In particular, the kind of positivism that’s self-refuting.
In what fashion is positivism self-refuting?
The proposition “only propositions that can be empirically tested are meaningful” cannot be empirically tested.
Its meaningless by its own epistemology.
Eppur si muove! It still works.
Works at what? Note that it,s not a synonym for science or empiricism , or the scientific method .
Pedantic complaint about language: moral realism simply says that moral claims do state facts, and at least some of them are true. It takes further assumptions (“internalism”) to claim that these moral facts are universally compelling in the sense of moving any intelligent being to action. (I personally believe the latter assumption to be nonsense, hence AGI is a really bad idea.)
Granted, I don’t know of any nice precise term for that position that all intelligent beings must necessarily do the right thing, possibly because it’s so ridiculous no philosopher would profess it publicly in such words. On the other hand, motivational internalism would seem to be very intuitive, judging by the pervasiveness of the view that AI doesn’t pose any risk.
Isn’t it called Convergence?
Are you under the impression that CEV advocates around here believe that all intelligent beings must necessarily do the right thing?
On the whole, confusion reigns, but there is a fairly consistent tendency to reject Intrinsic Motivation without argument.
What’s “Intrinsic Motivation”? The only hits for it on LW are about akrasia.
As in intrinsically motivating states and concpets
So, moral motivational internalism. Then I agree that we tend to reject it. For example, here. You can make it work by having “this motivates the person considering it” be incorporated into the definition of “right”, but that results in a relativist definition, and I don’t see any need for it anyway.
Motivational internalism may not be an obvious truth, but that doesn’t mean its falsehood is the default. I don’t see the relevance of the link.
So, basicly, what we call “terminal values”?
No, the idea of motivational internalism is that you can’t judge something as right or wrong without being motivated to pursue or avoid it. Like if the word “right” was short for “this thing matches my terminal values”.
The alternative is externalism, where “right” means {X, Y, Z} and we (some/most/all humans) are motivated to pursue it just because we like {X, Y, Z}.
Ah, OK. Thanks for explaining.
Does “Intrinsic Motivation” in this context entail that all intelligent beings must necessarily do the right thing?
If so, then I agree that we tend to reject it. As for “without argument”… do you mean you’ve read the local discussions of the topic and find them unconvincing? Or do you mean you believe it hasn’t been discussed at all?
If not, then I don’t know what you’re saying.
If you prefer to continue expressing yourself in gnomic utterances, that’s of course your choice, but I find it an unhelpful way to communicate and will tap out here if so.
If not, I’m
Little argument and none convincing.
Eh, maybe? I’ve seen “convergence thesis” thrown about on LW, but it’s hardly established terminology. Not sure it would be fair to use a phrase so easily confused with Bostrom’s much more reasonable Instrumental Convergence Thesis either. (Also, it has nothing to do with CEV so I don’t see the point of that link.)
From abstract reason or psychological facts, or physical facts, or a mixture.
There is a subject called economics. It tells you how to achieve certain goals, such as maximising GDP. It doesn’t do that by corresponding to a metaphysical Economics Object, it does that with a mixture of theoretical reasoning and examination of evidence.
There is a subject called ethics. It tells you how to achieve certain goals, such as maximising happiness....
Well there’s the problem: ethics does not automatically start out with a happiness-utilitarian goal. Lots of extent ethical systems use other terminal goals. For instance...
“Such as”
Sufficient rationality will tell you how to maximize any goal, once you can clearly define the goal.
Rationality is quite helpful for clarifying goals too.
Problem is, economics is not a science:
http://www.theatlantic.com/business/archive/2013/04/the-laws-of-economics-dont-exist/274901/
Of course economics doesn’t have the well-established laws of physical science: it wouldn’t be much of an analogy for ethics if it did.But having an epistemology that doens’t work very well is not the same as having an epistemology that requires non-natural entities.
The main problem with economics is not its descriptive, but its predictive power. Too many of economics’ calculations need to suppose that everyone will behave rationally, which regular people can’t be trusted to do. Same problem with politics.
Well, there’s a couple prongs to that. For one thing, it’s tagged as fiction in most people’s minds, as might be suggested by the fact that it’s easily described in trope. That’s bad enough by itself.
Probably more importantly, though, there’s a ferocious tendency to anthropomorphize this sort of thing, and you can’t really grok UFAI without burning a good bit of that tendency out of your head. Sure, we ourselves aren’t capital-F Friendly, but we’re a far cry yet from a paperclip maximizer or even most of the subtler failures of machine ethics; a jealous or capricious machine god is bad, but we’re talking Screwtape here, not Azathoth. HAL and Agent Smith are the villains of their stories, but they’re human in most of the ways that count.
You may also notice that we tend to win fictional robot wars.
Also, note that the tropes tend to work against people who say “we have a systematic proof that our design of AI will be Friendly”. In fact, in general the only way a fictional AI will turn out ‘friendly’ is if it is created entirely by accident—ANY fictional attempt to intentionally create a Friendly AI will result in an abomination, usually through some kind of “dick Genie” interpretation of its Friendliness rules.
Yeah. I think I’d consider that a form of backdoor anthropomorphization by way of vitalism, though. Since we tend to think of physically nonhuman intelligences as cognitively human, and since we tend to think of human ethics and cognition as something sacred and ineffable, fictional attempts to eff them tend to be written as crude morality plays.
Intelligence arising organically from a telephone exchange or an educational game or something doesn’t trigger the same taboos.
The currently fashionable descriptor is “metacontrarianism”—you might get better responses if you phrase your objection in that way.
(man, I LOVE when things go factorially N-meta)
I’m not actually sure who the metacontrarian is here.
Hence my delight in the factorial metaness.
Looks like you (emphasis mine):
and
You can be a contrarian with less drama perfectly well :-)
I would note that “we are all in the process of dying horribly” is actually a pretty dramatic situation. At the moment, actually, I’m not banking on ever seeing it: I think actual AI creation requires such expertise and has such extreme feasibility barriers that successfully building a functioning software-embodied optimization process tends to require such group efforts that someone thinks hard about what the goal system is.
Given that “we are all in the process of dying” is true for all living beings for as long as living beings existed, I don’t see anything dramatic in here. As to “horribly”, what is special about today’s “horror” compared to, say, a hundred years ago?
I hadn’t meant today. I had meant in the case of a UFAI getting loose. That’s one of those rare situations where you should consider yourself assuredly dead already and start considering how you’re going to kill the damn UFAI, whatever that costs you.
Whereas in the present day, I would not employ “nuke it from orbit; only way to be sure” solutions to, well, anything.
Thanks, Eli. You make some good points amidst the storm. :)
I think the scenario James elaborated was meant to be a fictional portrayal of a bad outcome that we should seek to avoid. That it was pasted without context may have given the impression that he actually supported such a strategy.
I mostly agree with your bullet points. Working toward cooperation and global unification, especially before things get ugly, is what I was suggesting in the opening post.
Even if uFAI would destroy its creators, people still have incentive to skimp on safety measures in an arms-race situation because they’re trading off some increased chance of winning against some increased chance of killing everyone. If winning the race is better than letting someone else win, then you’re willing to tolerate some increased risk of killing everyone. This is why I suggested promoting internationalist perspective as one way to improve the situation—because then individual countries would care less about winning the race.
BTW, it’s not clear that Clippy would kill us all. Like in any other struggle for power, a newly created Clippy might compromise with humans by keeping them alive and giving them some of what they want. This is especially likely if Clippy is risk averse.
Interesting. So there are backup safety strategies. That’s quite comforting to know, actually.
Oh thank God. I’d like to apologize for my behavior, but to be honest this community is oftentimes over my Poe’s Law Line where I can no longer actually tell if someone is acting out a fictional parody of a certain idea or actually believes in that idea.
Next time I guess I’ll just assign much more probability to the “this person is portraying a fictional hypothetical” notion.
Sorry, could you explain? I’m not seeing it. That is, I’m not seeing how increasing the probability that your victory equates with your own suicide is better than letting someone else just kill you. You’re dead either way.
No worries. :-)
Say that value(you win) = +4, value(others win) = +2, value(all die) = 0. If you skimp on safety measures for yourself, you can increase your probability of winning relative to others, and this is worth some increased chance of killing everyone. Let me know if you want further clarification. :) The final endpoint of this process will be a Nash equilibrium, as discussed in “Racing to the Precipice,” but what I described could be one step toward reaching that equilibrium.