Two things surprised me in your argument. One is that you seemed to assume that features of human ethics (which you attribute to our having evolved as social animals) would be universal in the sense that they would also apply to AIs which did not evolve and which aren’t necessarily social.
The second is that although you pay lip service to game theory, you don’t seem to be aware of any game theoretic research on ethics deeper than Axelrod(1984) and the Tit-for-Tat experiments. You ought to at least peruse Binsmore’s “Natural Justice”, even if you don’t want to plow through the two volumes of “Game Theory and the Social Contract”.
Being social is advantageous to any entity without terminal goals and advantageous to entities with terminal goals in most cases (primary exceptions being single goal entities, entities on the verge of achieving all of their terminal goals, and entities that are somehow guaranteed that they are and will remain far, far more powerful than everyone else). Humans evolved to be social because social was advantageous. A super-intelligent but non-evolved AGI will figure out that social is advantageous as well (except, obviously, in the very limited edge cases mentioned above).
Not quoting more research is not the same as being unaware of that research. I’ve read Binsmore—but how can I successfully bring it up when I can’t even get acceptance of Axelrod? It’s like trying to teach multiplication while addition is still a problem. I really should read GT&tSC. It’s been on my reading list since I’ve tasked myself with writing something in response to Rawls’ corpus. I just haven’t gotten around to it.
I have presented further works on the same subject at BICA ’09 and AGI ‘10 (with a really fun second presentation at AGI ’10 here) but haven’t advanced the game theory portion at all (unfortunately). My focus has recently shifted radically though and going back to game theory could help that tremendously. Thanks.
Being social is advantageous to any entity without terminal goals [your emphasis]
I can’t accept this. Many animals are not social, or are social only to the extent of practicing parental care.
A super-intelligent but non-evolved AGI will figure out that social is advantageous as well.
Only if it is actually advantageous to them (it?). Your claim would be much more convincing if you could provide examples of what AIs might gain by social interaction with humans, and why the AI could not achieve the same benefits with less risk and effort by exterminating or enslaving us. Without such examples, your bare assertions are completely unconvincing.
Please note that as humans evolved to their current elevated moral plane they occasionally found extermination and enslavement to be more tempting solutions to their social problems than reciprocity. In fact, enslavement is a form of reciprocity—it is one possible solution to a bargaining problem as in Nash(1953). A solution in which one bargainer has access to much better threats than the other.
Many animals are not smart enough to be social. We are talking about a super-intelligent AGI here. I’ve given several presentations at conferences including AGI-09 with J Storrs Hall showing that animals are sociable to the extent that their cognitive apparatus can support it (not to mention the incredible socialness of bees, termites, etc.)
What do we gain from social interactions with dogs? Do we honestly suffer no losses when we mindlessly trash the rain forests? Examples to support my “bare assumptions” are EASY to come by (but thanks for asking—I just wish that other people here would give me examples when I ask).
Enslavement is an excellent short-term solution; however, in the long-term, it is virtually always a net negative to the system as a whole (i.e. it is selfish and stupid when viewed from the longest term—and moreso the more organized and inter-related the system is). Once again, we are talking about a super-intelligence, not short-sighted, stupid humans (who are, nonetheless, inarguably getting better and better with time).
What do we gain from social interactions with dogs? … Examples to support my “bare assumptions” are EASY to come by.
Happy to hear that. Because it then becomes reasonable to assume that you would not find it burdensome to share those examples. We are talking about benefits that a powerful AI would derive from social interactions with humans.
I hope you have something more than the implied analogy of the human-canine relationship. Because there are many other species just as intelligent as dogs with which we humans do not share quite so reciprocal a relationship. And, perhaps it is just my pride, but I don’t really think that I would appreciate being treated like a dog by my AI master. ETA: And I don’t know of any human dog-lovers who keep 6 billion pets.
Because there are many other species just as intelligent as dogs with which we humans do not share quite so reciprocal a relationship.
Absolutely. Because dogs cooperate with us and we with them and the other species don’t.
And, perhaps it is just my pride, but I don’t really think that I would appreciate being treated like a dog by my AI master.
And immediately the human prejudice comes out. We have terrible behavior when we’re on the top of the pile and expects others to have it as well. It’s almost exactly the same as when people complain bitterly when they’re oppressed and then, when they are on top, they oppress others even worse.
What is wrong with the human-canine analogy (which I thought I did more than imply) is the baggage that you are bringing to that relationship. Both parties benefit from the relationship. The dog benefits less from that relationship than you would benefit from an AGI relationship because the dog is less competent and intelligent than you are AND because the dog generally likes the treatment that it receives (whereas you would be unhappy with similar treatment).
Dogs are THE BEST analogy because they are the closest existing example to what most people are willing to concede is likely to be our relationship with a super-advanced AGI.
Oh, and dogs don’t really have a clue as to what they do for us, so why do you expect me to be able to come up with what we will do for an advanced AGI? If we’re willing to cooperate, there will be plenty for us to do of value that will fulfill our goals as well. We just have to avoid being too paranoid and short-sighted to see it.
earthworm—three orders of magnitude--> small lizard—three orders of magnitude--> dog—three orders of magnitude--> human—thirty orders of magnitude--> weakly superhuman AGI—several thousand orders of magnitude--> strong AI
If a recursively self-improving process stopped just far enough above us to consider us pets and did so, I would seriously question whether it was genuinely recursive, or if it was just gains from debugging and streamlining human thought process. ie, I could see a self-modifying transhuman acting in the manner you describe. But not an artificial intelligence, not unless it was very carefully designed.
entities that are somehow guaranteed that they are and will remain far, far more powerful than everyone else
And you don’t think a self-improving AI will ever fall into this category? Hell, if you gave a human the ability to run billions of simulations per second to study how their decisions would turn out, they’d be able to take over the world and “remain far, far more powerful” than everyone else. (If they were actually more intelligent, and not just faster, even more so.)
Your so-called “limited edge case” is the main case being discussed: superhuman intelligence. (The problem of single-goal entities is of course also discussed here; see the idea of a “paper-clip maximizer”, for example.)
In short, you seem to be saying that we shouldn’t worry about those “edge” cases because in all non-”edge” cases, things work out fine. That’s like saying we shouldn’t worry about having fire departments or constructing homes according to a fire code, because a fire is an “edge” case, and normally buildings don’t burn down.
Even if you were to make such an argument, it makes little sense to propose it at a meeting of the fire council. ;-)
It may be true that mostly, fires don’t happen. However, it’s also true that if you don’t build the buildings with fire prevention (and especially, preventing the spread of fires) in mind, then, sooner or later, your whole city burns down. Because at that point, it only takes one fire to do it.
entities that are somehow guaranteed that they are and will remain far, far more powerful than everyone else
And you don’t think a self-improving AI will ever fall into this category?
You mean “somehow guaranteed ”? No, I don’t believe that a self-improving AI will ever fall into this category. It might decide to believe it—which would be very dangerous for us—but, no, I don’t believe that it is likely to truly find such a guarantee. Further, given the VERY minimal cost (if any) of cooperating with a cooperating entity, an AI would be human-foolish to take the stupid short-sighted shortcut of trashing us for no reason—since it certainly is an existential risk for IT that something bigger and smarter would take exception to such a diversity-decreasing act.
MORE IMPORTANTLY—you dropped the fact that the AI already has to have one flaw (terminal goals) before this second aspect could possibly become a problem.
Fire is not an “edge” case. The probability of a building catching fire in a city any given day is VERY high. But that is irrelevant because . . . .
you ALWAYS worry about edge cases. In this case, though, if you are aware of them and plan/prepare against them—they are AVOIDABLE edge cases (more so than the city burning down even if you have fire prevention c.f. Chicago & Mrs. O’Leary’s cow).
an AI would be human-foolish to take the stupid short-sighted shortcut of trashing us for no reason
You don’t seem to understand how basic reasoning works (by LW standards). AFAICT, you are both privileging your hypothesis, and not weighing any evidence.
(Heck, you’re not even stating any evidence, only relying on repeated assertion of your framing of the situation.)
You still haven’t responded, for example, to my previous point about human-bacterium empathy. We don’t have empathy for bacteria, in part because we see them as interchangeable and easily replaced. If for some reason we want some more E. coli, we can just culture some.
In the same way, a superhuman intelligence that anticipates a possible future use for human beings, could always just keep our DNA on file… with a modification or two to make us more pliable.
Your entire argument is based on an enormous blind spot from your genetic heritage: you think an AI would inherently see you as, well, “human”, when out of the space of all possible minds, the odds of a given AI seeing you as worth bothering with are negligible at best. You simply don’t see this, because your built-in machinery for imagining minds automatically imagines human minds—even when you try to make it not do so.
Hell, the human-bacterium analogy is a perfect example: I’m using that example specifically because it’s a human way of thinking, even though it’s unlikely to match the utter lack of caring with which an arbitrary AGI is likely to view human beings. It’s wrong to even think of it as “viewing”, because that supposes a human model.
AI’s are not humans, unless they’re built to be humans, and the odds of them being human by accident are negligible.
Remember: evolution is happy to have elephants slowly starve to death when they get old, and to have animals that die struggling and painfully in the act of mating. Arbitrary optimization processes do not have human values.
Here I was assuming that PJ had integrated Descartes and Zen and was trying to understand the deep wisdom behind the koan.
The scary thing is, if I engage “extracting wisdom from koan” mode I can actually feel “you are both your hypothesis, and not weighing any evidence” fitting in neatly with actual insights that fit within PJ’s area of expertise. +1 to pattern matching on noise!
Yes, don’t know how that got deleted, because I saw it in there shortly before posting. My copy of Firefox sometimes does odd things during text editing.
So give me some examples. Cooperation is a non-zero-sum game that continues adding utility the longer it goes on. Do you deny that this is the case?
Oh, wait, you’re the same guy who, whenever asked to back up his statements, never does.
Please support me and your community by doing more than throwing cryptic opinionated darts and then refusing to elaborate. You’re only wasting everyone’s time and acting as a drag on the community.
The Prisoner’s Dilemma. Classic, classic example where cooperation has a non-minimal cost—ie, the risk that they will defect against you, multiplied by the probability that they will defect, is the cost of cooperating.
VERY minimal cost (if any) of cooperating with a cooperating entity
And in the Prisoner’s Dilemma, if you somehow specify the other entity is cooperating, then cooperating with a cooperating entity carries a cost still: the difference between “both cooperate” and “you defect against their cooperate” is the cost of cooperating there.
throwing cryptic opinionated darts
If your argument rested on some mathematical concepts, and one of them was an equation that you derived incorrectly, and wedrifid pointed that out, would he still be throwing darts? He wasn’t telling you that you were wrong because he hates you, or because he enjoys ruining peoples’ time on this blog, or any other sadistic personality trait, he was pointing out the flaw because it was flawed.
Cooperation is a non-zero-sum game that continues adding utility the longer it goes on. Do you deny that this is the case?
That is sometimes referred to as “mutually beneficial” cooperation:
Co-operation or co-operative behaviours are terms used to describe behaviours by organisms which are beneficial to other organisms, and are selected for on that basis. Under this definition, altruism is a form of co-operation in which there is no direct benefit to the actor (the organism carrying out the behaviour). Co-operative behaviour in which there is a direct benefit to the actor as well as the recipient can be termed “mutually beneficial”.
Two things surprised me in your argument. One is that you seemed to assume that features of human ethics (which you attribute to our having evolved as social animals) would be universal in the sense that they would also apply to AIs which did not evolve and which aren’t necessarily social.
The second is that although you pay lip service to game theory, you don’t seem to be aware of any game theoretic research on ethics deeper than Axelrod(1984) and the Tit-for-Tat experiments. You ought to at least peruse Binsmore’s “Natural Justice”, even if you don’t want to plow through the two volumes of “Game Theory and the Social Contract”.
Being social is advantageous to any entity without terminal goals and advantageous to entities with terminal goals in most cases (primary exceptions being single goal entities, entities on the verge of achieving all of their terminal goals, and entities that are somehow guaranteed that they are and will remain far, far more powerful than everyone else). Humans evolved to be social because social was advantageous. A super-intelligent but non-evolved AGI will figure out that social is advantageous as well (except, obviously, in the very limited edge cases mentioned above).
Not quoting more research is not the same as being unaware of that research. I’ve read Binsmore—but how can I successfully bring it up when I can’t even get acceptance of Axelrod? It’s like trying to teach multiplication while addition is still a problem. I really should read GT&tSC. It’s been on my reading list since I’ve tasked myself with writing something in response to Rawls’ corpus. I just haven’t gotten around to it.
I have presented further works on the same subject at BICA ’09 and AGI ‘10 (with a really fun second presentation at AGI ’10 here) but haven’t advanced the game theory portion at all (unfortunately). My focus has recently shifted radically though and going back to game theory could help that tremendously. Thanks.
I can’t accept this. Many animals are not social, or are social only to the extent of practicing parental care.
Only if it is actually advantageous to them (it?). Your claim would be much more convincing if you could provide examples of what AIs might gain by social interaction with humans, and why the AI could not achieve the same benefits with less risk and effort by exterminating or enslaving us. Without such examples, your bare assertions are completely unconvincing.
Please note that as humans evolved to their current elevated moral plane they occasionally found extermination and enslavement to be more tempting solutions to their social problems than reciprocity. In fact, enslavement is a form of reciprocity—it is one possible solution to a bargaining problem as in Nash(1953). A solution in which one bargainer has access to much better threats than the other.
Many animals are not smart enough to be social. We are talking about a super-intelligent AGI here. I’ve given several presentations at conferences including AGI-09 with J Storrs Hall showing that animals are sociable to the extent that their cognitive apparatus can support it (not to mention the incredible socialness of bees, termites, etc.)
What do we gain from social interactions with dogs? Do we honestly suffer no losses when we mindlessly trash the rain forests? Examples to support my “bare assumptions” are EASY to come by (but thanks for asking—I just wish that other people here would give me examples when I ask).
Enslavement is an excellent short-term solution; however, in the long-term, it is virtually always a net negative to the system as a whole (i.e. it is selfish and stupid when viewed from the longest term—and moreso the more organized and inter-related the system is). Once again, we are talking about a super-intelligence, not short-sighted, stupid humans (who are, nonetheless, inarguably getting better and better with time).
Happy to hear that. Because it then becomes reasonable to assume that you would not find it burdensome to share those examples. We are talking about benefits that a powerful AI would derive from social interactions with humans.
I hope you have something more than the implied analogy of the human-canine relationship. Because there are many other species just as intelligent as dogs with which we humans do not share quite so reciprocal a relationship. And, perhaps it is just my pride, but I don’t really think that I would appreciate being treated like a dog by my AI master. ETA: And I don’t know of any human dog-lovers who keep 6 billion pets.
Absolutely. Because dogs cooperate with us and we with them and the other species don’t.
And immediately the human prejudice comes out. We have terrible behavior when we’re on the top of the pile and expects others to have it as well. It’s almost exactly the same as when people complain bitterly when they’re oppressed and then, when they are on top, they oppress others even worse.
What is wrong with the human-canine analogy (which I thought I did more than imply) is the baggage that you are bringing to that relationship. Both parties benefit from the relationship. The dog benefits less from that relationship than you would benefit from an AGI relationship because the dog is less competent and intelligent than you are AND because the dog generally likes the treatment that it receives (whereas you would be unhappy with similar treatment).
Dogs are THE BEST analogy because they are the closest existing example to what most people are willing to concede is likely to be our relationship with a super-advanced AGI.
Oh, and dogs don’t really have a clue as to what they do for us, so why do you expect me to be able to come up with what we will do for an advanced AGI? If we’re willing to cooperate, there will be plenty for us to do of value that will fulfill our goals as well. We just have to avoid being too paranoid and short-sighted to see it.
The scale is all out.
earthworm—three orders of magnitude--> small lizard—three orders of magnitude--> dog—three orders of magnitude--> human—thirty orders of magnitude--> weakly superhuman AGI—several thousand orders of magnitude--> strong AI
If a recursively self-improving process stopped just far enough above us to consider us pets and did so, I would seriously question whether it was genuinely recursive, or if it was just gains from debugging and streamlining human thought process. ie, I could see a self-modifying transhuman acting in the manner you describe. But not an artificial intelligence, not unless it was very carefully designed.
Stop wasting our time.
Hmm. What do you mean by an “entity without terminal goals”. Would a rock qualify?
Perhaps not.
No. A rock is not an entity.
Right. Many people here use the term “terminal” in the following sense (in this kind of context):
http://en.wikipedia.org/wiki/Terminalvalue(philosophy))
However, interpreting your comment in the light of such a definition apparently makes little sense.
So—presumably you meant something else—but what?
link)
(Escaping closing parenthesis to ensure the link syntax is not prematurely closed.)
And you don’t think a self-improving AI will ever fall into this category? Hell, if you gave a human the ability to run billions of simulations per second to study how their decisions would turn out, they’d be able to take over the world and “remain far, far more powerful” than everyone else. (If they were actually more intelligent, and not just faster, even more so.)
Your so-called “limited edge case” is the main case being discussed: superhuman intelligence. (The problem of single-goal entities is of course also discussed here; see the idea of a “paper-clip maximizer”, for example.)
In short, you seem to be saying that we shouldn’t worry about those “edge” cases because in all non-”edge” cases, things work out fine. That’s like saying we shouldn’t worry about having fire departments or constructing homes according to a fire code, because a fire is an “edge” case, and normally buildings don’t burn down.
Even if you were to make such an argument, it makes little sense to propose it at a meeting of the fire council. ;-)
It may be true that mostly, fires don’t happen. However, it’s also true that if you don’t build the buildings with fire prevention (and especially, preventing the spread of fires) in mind, then, sooner or later, your whole city burns down. Because at that point, it only takes one fire to do it.
You mean “somehow guaranteed ”? No, I don’t believe that a self-improving AI will ever fall into this category. It might decide to believe it—which would be very dangerous for us—but, no, I don’t believe that it is likely to truly find such a guarantee. Further, given the VERY minimal cost (if any) of cooperating with a cooperating entity, an AI would be human-foolish to take the stupid short-sighted shortcut of trashing us for no reason—since it certainly is an existential risk for IT that something bigger and smarter would take exception to such a diversity-decreasing act.
MORE IMPORTANTLY—you dropped the fact that the AI already has to have one flaw (terminal goals) before this second aspect could possibly become a problem.
Fire is not an “edge” case. The probability of a building catching fire in a city any given day is VERY high. But that is irrelevant because . . . .
you ALWAYS worry about edge cases. In this case, though, if you are aware of them and plan/prepare against them—they are AVOIDABLE edge cases (more so than the city burning down even if you have fire prevention c.f. Chicago & Mrs. O’Leary’s cow).
You don’t seem to understand how basic reasoning works (by LW standards). AFAICT, you are both privileging your hypothesis, and not weighing any evidence.
(Heck, you’re not even stating any evidence, only relying on repeated assertion of your framing of the situation.)
You still haven’t responded, for example, to my previous point about human-bacterium empathy. We don’t have empathy for bacteria, in part because we see them as interchangeable and easily replaced. If for some reason we want some more E. coli, we can just culture some.
In the same way, a superhuman intelligence that anticipates a possible future use for human beings, could always just keep our DNA on file… with a modification or two to make us more pliable.
Your entire argument is based on an enormous blind spot from your genetic heritage: you think an AI would inherently see you as, well, “human”, when out of the space of all possible minds, the odds of a given AI seeing you as worth bothering with are negligible at best. You simply don’t see this, because your built-in machinery for imagining minds automatically imagines human minds—even when you try to make it not do so.
Hell, the human-bacterium analogy is a perfect example: I’m using that example specifically because it’s a human way of thinking, even though it’s unlikely to match the utter lack of caring with which an arbitrary AGI is likely to view human beings. It’s wrong to even think of it as “viewing”, because that supposes a human model.
AI’s are not humans, unless they’re built to be humans, and the odds of them being human by accident are negligible.
Remember: evolution is happy to have elephants slowly starve to death when they get old, and to have animals that die struggling and painfully in the act of mating. Arbitrary optimization processes do not have human values.
Stop thinking “intellect” (i.e. human) and start thinking “mechanical optimization process”.
[edit to add: “privileging”, which somehow got eaten while writing the original comment]
you are both privileging your hypothesis ?
Here I was assuming that PJ had integrated Descartes and Zen and was trying to understand the deep wisdom behind the koan.
The scary thing is, if I engage “extracting wisdom from koan” mode I can actually feel “you are both your hypothesis, and not weighing any evidence” fitting in neatly with actual insights that fit within PJ’s area of expertise. +1 to pattern matching on noise!
Even scarier thought: suppose that what we think of as intelligence or creativity consists, in simple fact, of pattern matching on random noise? ;-)
Yes, don’t know how that got deleted, because I saw it in there shortly before posting. My copy of Firefox sometimes does odd things during text editing.
This premise is VERY flawed.
So give me some examples. Cooperation is a non-zero-sum game that continues adding utility the longer it goes on. Do you deny that this is the case?
Oh, wait, you’re the same guy who, whenever asked to back up his statements, never does.
Please support me and your community by doing more than throwing cryptic opinionated darts and then refusing to elaborate. You’re only wasting everyone’s time and acting as a drag on the community.
The Prisoner’s Dilemma. Classic, classic example where cooperation has a non-minimal cost—ie, the risk that they will defect against you, multiplied by the probability that they will defect, is the cost of cooperating.
And in the Prisoner’s Dilemma, if you somehow specify the other entity is cooperating, then cooperating with a cooperating entity carries a cost still: the difference between “both cooperate” and “you defect against their cooperate” is the cost of cooperating there.
If your argument rested on some mathematical concepts, and one of them was an equation that you derived incorrectly, and wedrifid pointed that out, would he still be throwing darts? He wasn’t telling you that you were wrong because he hates you, or because he enjoys ruining peoples’ time on this blog, or any other sadistic personality trait, he was pointing out the flaw because it was flawed.
That is sometimes referred to as “mutually beneficial” cooperation:
http://en.wikipedia.org/wiki/Co-operation_(evolution)