entities that are somehow guaranteed that they are and will remain far, far more powerful than everyone else
And you don’t think a self-improving AI will ever fall into this category? Hell, if you gave a human the ability to run billions of simulations per second to study how their decisions would turn out, they’d be able to take over the world and “remain far, far more powerful” than everyone else. (If they were actually more intelligent, and not just faster, even more so.)
Your so-called “limited edge case” is the main case being discussed: superhuman intelligence. (The problem of single-goal entities is of course also discussed here; see the idea of a “paper-clip maximizer”, for example.)
In short, you seem to be saying that we shouldn’t worry about those “edge” cases because in all non-”edge” cases, things work out fine. That’s like saying we shouldn’t worry about having fire departments or constructing homes according to a fire code, because a fire is an “edge” case, and normally buildings don’t burn down.
Even if you were to make such an argument, it makes little sense to propose it at a meeting of the fire council. ;-)
It may be true that mostly, fires don’t happen. However, it’s also true that if you don’t build the buildings with fire prevention (and especially, preventing the spread of fires) in mind, then, sooner or later, your whole city burns down. Because at that point, it only takes one fire to do it.
entities that are somehow guaranteed that they are and will remain far, far more powerful than everyone else
And you don’t think a self-improving AI will ever fall into this category?
You mean “somehow guaranteed ”? No, I don’t believe that a self-improving AI will ever fall into this category. It might decide to believe it—which would be very dangerous for us—but, no, I don’t believe that it is likely to truly find such a guarantee. Further, given the VERY minimal cost (if any) of cooperating with a cooperating entity, an AI would be human-foolish to take the stupid short-sighted shortcut of trashing us for no reason—since it certainly is an existential risk for IT that something bigger and smarter would take exception to such a diversity-decreasing act.
MORE IMPORTANTLY—you dropped the fact that the AI already has to have one flaw (terminal goals) before this second aspect could possibly become a problem.
Fire is not an “edge” case. The probability of a building catching fire in a city any given day is VERY high. But that is irrelevant because . . . .
you ALWAYS worry about edge cases. In this case, though, if you are aware of them and plan/prepare against them—they are AVOIDABLE edge cases (more so than the city burning down even if you have fire prevention c.f. Chicago & Mrs. O’Leary’s cow).
an AI would be human-foolish to take the stupid short-sighted shortcut of trashing us for no reason
You don’t seem to understand how basic reasoning works (by LW standards). AFAICT, you are both privileging your hypothesis, and not weighing any evidence.
(Heck, you’re not even stating any evidence, only relying on repeated assertion of your framing of the situation.)
You still haven’t responded, for example, to my previous point about human-bacterium empathy. We don’t have empathy for bacteria, in part because we see them as interchangeable and easily replaced. If for some reason we want some more E. coli, we can just culture some.
In the same way, a superhuman intelligence that anticipates a possible future use for human beings, could always just keep our DNA on file… with a modification or two to make us more pliable.
Your entire argument is based on an enormous blind spot from your genetic heritage: you think an AI would inherently see you as, well, “human”, when out of the space of all possible minds, the odds of a given AI seeing you as worth bothering with are negligible at best. You simply don’t see this, because your built-in machinery for imagining minds automatically imagines human minds—even when you try to make it not do so.
Hell, the human-bacterium analogy is a perfect example: I’m using that example specifically because it’s a human way of thinking, even though it’s unlikely to match the utter lack of caring with which an arbitrary AGI is likely to view human beings. It’s wrong to even think of it as “viewing”, because that supposes a human model.
AI’s are not humans, unless they’re built to be humans, and the odds of them being human by accident are negligible.
Remember: evolution is happy to have elephants slowly starve to death when they get old, and to have animals that die struggling and painfully in the act of mating. Arbitrary optimization processes do not have human values.
Here I was assuming that PJ had integrated Descartes and Zen and was trying to understand the deep wisdom behind the koan.
The scary thing is, if I engage “extracting wisdom from koan” mode I can actually feel “you are both your hypothesis, and not weighing any evidence” fitting in neatly with actual insights that fit within PJ’s area of expertise. +1 to pattern matching on noise!
Yes, don’t know how that got deleted, because I saw it in there shortly before posting. My copy of Firefox sometimes does odd things during text editing.
So give me some examples. Cooperation is a non-zero-sum game that continues adding utility the longer it goes on. Do you deny that this is the case?
Oh, wait, you’re the same guy who, whenever asked to back up his statements, never does.
Please support me and your community by doing more than throwing cryptic opinionated darts and then refusing to elaborate. You’re only wasting everyone’s time and acting as a drag on the community.
The Prisoner’s Dilemma. Classic, classic example where cooperation has a non-minimal cost—ie, the risk that they will defect against you, multiplied by the probability that they will defect, is the cost of cooperating.
VERY minimal cost (if any) of cooperating with a cooperating entity
And in the Prisoner’s Dilemma, if you somehow specify the other entity is cooperating, then cooperating with a cooperating entity carries a cost still: the difference between “both cooperate” and “you defect against their cooperate” is the cost of cooperating there.
throwing cryptic opinionated darts
If your argument rested on some mathematical concepts, and one of them was an equation that you derived incorrectly, and wedrifid pointed that out, would he still be throwing darts? He wasn’t telling you that you were wrong because he hates you, or because he enjoys ruining peoples’ time on this blog, or any other sadistic personality trait, he was pointing out the flaw because it was flawed.
Cooperation is a non-zero-sum game that continues adding utility the longer it goes on. Do you deny that this is the case?
That is sometimes referred to as “mutually beneficial” cooperation:
Co-operation or co-operative behaviours are terms used to describe behaviours by organisms which are beneficial to other organisms, and are selected for on that basis. Under this definition, altruism is a form of co-operation in which there is no direct benefit to the actor (the organism carrying out the behaviour). Co-operative behaviour in which there is a direct benefit to the actor as well as the recipient can be termed “mutually beneficial”.
And you don’t think a self-improving AI will ever fall into this category? Hell, if you gave a human the ability to run billions of simulations per second to study how their decisions would turn out, they’d be able to take over the world and “remain far, far more powerful” than everyone else. (If they were actually more intelligent, and not just faster, even more so.)
Your so-called “limited edge case” is the main case being discussed: superhuman intelligence. (The problem of single-goal entities is of course also discussed here; see the idea of a “paper-clip maximizer”, for example.)
In short, you seem to be saying that we shouldn’t worry about those “edge” cases because in all non-”edge” cases, things work out fine. That’s like saying we shouldn’t worry about having fire departments or constructing homes according to a fire code, because a fire is an “edge” case, and normally buildings don’t burn down.
Even if you were to make such an argument, it makes little sense to propose it at a meeting of the fire council. ;-)
It may be true that mostly, fires don’t happen. However, it’s also true that if you don’t build the buildings with fire prevention (and especially, preventing the spread of fires) in mind, then, sooner or later, your whole city burns down. Because at that point, it only takes one fire to do it.
You mean “somehow guaranteed ”? No, I don’t believe that a self-improving AI will ever fall into this category. It might decide to believe it—which would be very dangerous for us—but, no, I don’t believe that it is likely to truly find such a guarantee. Further, given the VERY minimal cost (if any) of cooperating with a cooperating entity, an AI would be human-foolish to take the stupid short-sighted shortcut of trashing us for no reason—since it certainly is an existential risk for IT that something bigger and smarter would take exception to such a diversity-decreasing act.
MORE IMPORTANTLY—you dropped the fact that the AI already has to have one flaw (terminal goals) before this second aspect could possibly become a problem.
Fire is not an “edge” case. The probability of a building catching fire in a city any given day is VERY high. But that is irrelevant because . . . .
you ALWAYS worry about edge cases. In this case, though, if you are aware of them and plan/prepare against them—they are AVOIDABLE edge cases (more so than the city burning down even if you have fire prevention c.f. Chicago & Mrs. O’Leary’s cow).
You don’t seem to understand how basic reasoning works (by LW standards). AFAICT, you are both privileging your hypothesis, and not weighing any evidence.
(Heck, you’re not even stating any evidence, only relying on repeated assertion of your framing of the situation.)
You still haven’t responded, for example, to my previous point about human-bacterium empathy. We don’t have empathy for bacteria, in part because we see them as interchangeable and easily replaced. If for some reason we want some more E. coli, we can just culture some.
In the same way, a superhuman intelligence that anticipates a possible future use for human beings, could always just keep our DNA on file… with a modification or two to make us more pliable.
Your entire argument is based on an enormous blind spot from your genetic heritage: you think an AI would inherently see you as, well, “human”, when out of the space of all possible minds, the odds of a given AI seeing you as worth bothering with are negligible at best. You simply don’t see this, because your built-in machinery for imagining minds automatically imagines human minds—even when you try to make it not do so.
Hell, the human-bacterium analogy is a perfect example: I’m using that example specifically because it’s a human way of thinking, even though it’s unlikely to match the utter lack of caring with which an arbitrary AGI is likely to view human beings. It’s wrong to even think of it as “viewing”, because that supposes a human model.
AI’s are not humans, unless they’re built to be humans, and the odds of them being human by accident are negligible.
Remember: evolution is happy to have elephants slowly starve to death when they get old, and to have animals that die struggling and painfully in the act of mating. Arbitrary optimization processes do not have human values.
Stop thinking “intellect” (i.e. human) and start thinking “mechanical optimization process”.
[edit to add: “privileging”, which somehow got eaten while writing the original comment]
you are both privileging your hypothesis ?
Here I was assuming that PJ had integrated Descartes and Zen and was trying to understand the deep wisdom behind the koan.
The scary thing is, if I engage “extracting wisdom from koan” mode I can actually feel “you are both your hypothesis, and not weighing any evidence” fitting in neatly with actual insights that fit within PJ’s area of expertise. +1 to pattern matching on noise!
Even scarier thought: suppose that what we think of as intelligence or creativity consists, in simple fact, of pattern matching on random noise? ;-)
Yes, don’t know how that got deleted, because I saw it in there shortly before posting. My copy of Firefox sometimes does odd things during text editing.
This premise is VERY flawed.
So give me some examples. Cooperation is a non-zero-sum game that continues adding utility the longer it goes on. Do you deny that this is the case?
Oh, wait, you’re the same guy who, whenever asked to back up his statements, never does.
Please support me and your community by doing more than throwing cryptic opinionated darts and then refusing to elaborate. You’re only wasting everyone’s time and acting as a drag on the community.
The Prisoner’s Dilemma. Classic, classic example where cooperation has a non-minimal cost—ie, the risk that they will defect against you, multiplied by the probability that they will defect, is the cost of cooperating.
And in the Prisoner’s Dilemma, if you somehow specify the other entity is cooperating, then cooperating with a cooperating entity carries a cost still: the difference between “both cooperate” and “you defect against their cooperate” is the cost of cooperating there.
If your argument rested on some mathematical concepts, and one of them was an equation that you derived incorrectly, and wedrifid pointed that out, would he still be throwing darts? He wasn’t telling you that you were wrong because he hates you, or because he enjoys ruining peoples’ time on this blog, or any other sadistic personality trait, he was pointing out the flaw because it was flawed.
That is sometimes referred to as “mutually beneficial” cooperation:
http://en.wikipedia.org/wiki/Co-operation_(evolution)