I will admit to an estimate higher than 95% that humanity or its uploads will survive the next hundred years. Many of the “apocalyptic” scenarios people are concerned about seem unlikely to wipe out all of humanity; so long as we have a breeding population, we can recover.
No significant risk of unFriendly AI (especially since you apparently consider uploading within 100 years plausible)? Nanotech war? Even engineered disease? I’m surprised.
The comment appears to me to be saying there is no significant risk of wiping out all of humanity, not that there is no significant risk of any of the dangers you describe causing significant harm.
I think an unfriendly AI is somewhat likely for example but put a very low probability on an unfriendly AI completely wiping out humanity. The consequences could be quite unpleasant and worth working to avoid but I don’t think it’s an existential threat with any significant probability.
That’s a very strange perspective. Other threats are good in that they are stupid, so they won’t find you if you colonize space or live on an isolated island, or have a lucky combination of genes, or figure out a way to actively outsmart them, etc. Stupid existential risks won’t methodically exterminate every human, and so there is a chance for recovery. Unfriendly AI, on the other hand, won’t go away, and you can’t hide from it on another planet. (Indifference works this way too, it’s the application of power indifferent to humankind that is methodical, e.g. Paperclip AI.)
I think it is worth considering the number of species to which humanity is largely indifferent which are extinct as a result of humanity optimizing other criteria
Humans satisfice, and not very well at that compared to what an AGI could do. If we effectively optimized for… almost any goal not referring to fieldmice… fieldmice would be extinct.
Humanity is beautiful. A significantly more intelligent AI will love us more perfectly (no: more truly) than we love a field mouse. (It is an intermediately stupid AI that I would worry about.)
Later edit: If you’re interested in reading past group discussion on the topic of how superintelligence does not imply supermorality, search “surface analogies” and “supermorality”.
Affective rhetoric. It seems like you are reasoning by surface analogies that don’t apply, anthropomorphizing AIs without realizing that, thinking that you’ve successfully abstracted away all the human-specific (and irrelevant!) things. Unless you are condensing a deeper model, which you’d need to present in more detail to discuss, you just need to learn more about the subject before drawing any strong conclusions. Read up on the Yudkowsky’s posts. (For this comment in particular: see That Tiny Note of Discord and dependencies, although that’s far from an ideal first post on the subject.)
Yes, I understand that this is not religion and all positions will have to be argued and defended in due time. I am merely declaring my position. I do find it really fascinating that, in the first stages of drafting this new map, we begin by drawing lines in the sand…
It’s more that the counterargument against your position was covered, at great length, and then covered some more, on OB by Yudkowsky, the person that most of are here because we respect.
If you’re going to take a stand for something that most people here have already read very persuasive arguments against, I don’t think it’s unreasonable to expect more than just a position statement (and an emotionally-loaded one, at that).
I meant no disrespect. (Eliezer has 661 posts on OB.) I do appreciate your direction/correction. I didn’t mean to take a stand against.
(Sigh.) I have no positions, no beliefs, prior to what I might learn from Eliezer.
So the idea is that a unique, complex thing may not necessarily have an appreciation for another unique complexity? Unless appreciating unique complexity has a mathematical basis.
brynema, “disrespect” isn’t at all the the right axis for understanding why your last couple comments weren’t helpful. (I’m not attacking you here; LW is an unusual place, and understanding how to usefully contribute takes time. You’ve been doing well.) The trouble with your last two comments is mostly:
Comments on LW should aspire to rationality. As part of this aspiration, we basically shouldn’t have “positions” on issues we haven’t thought much about; the beliefs we share here should be evidence-based best-guesses about the future, not clothes to decorate ourselves with.
Many places encourage people to make up and share “beliefs”, because any person’s beliefs are as good as any other’s and it’s good to express oneself, or something like that. Those norms are not useful toward arriving at truth, at least not compared to what we usually manage on LW. Not even if people follow their made-up “beliefs” with evidence created to support their conclusions; nor even if evidence or intuitions play some role in the initial forming of beliefs.
This is particularly true in cases where the subjects are difficult technical problems that some in the community have specialized in and thought carefully about; declaring positions there is kind of like approaching a physicist, without knowledge of physics, and announcing your “position” on how atoms hold together. (Though less so, since AI is less well-grounded than physics.)
AI risks are a particularly difficult subject about which to have useful conversation, mostly because there is little data to help keep conversation from veering off into nonsense-land. So it makes sense, in discussing AI risks and other slippery topics, to have lower tolerance for folks making up positions.
Also, yes, these particular positions have been discussed and have proven un-workable in pretty exhaustive detail.
As to the object-level issue concerning possible minds, I wrote an answer in the welcome thread, on the theory that, if we want to talk about AI or other prerequisite-requiring topics on LW, we should probably get in the habit of taking “already discussed to death” questions to the welcome thread, where they won’t clutter mainline discussion. Please don’t be offended by this, though; I value your presence here, and you had no real way of knowing this had already been discussed.
I’ve spent some time working through my emotional responses and intellectual defenses to the posts above. I would like to make some observations:
(1) I’m disappointed that even as rationalists, while you were able to recognize that I had committed some transgression, you were not able to identify it precisely and power was used (authority, shaming) instead of the truth to broadly punish me.
(2) My mistake was not in asserting something false. This happens all the time here and people usually respond more rationally.
(3) My transgression was using the emotionally loaded word “love”. (So SoulessAutomaton actually came close.) The word in this context is taboo for a good reason—I will try to explain but perhaps I will fail: while I believe in love, I should not put the belief in those terms because invoking the word is dark art manipulation; the whole point of rationality is to find a better vocabulary for explaining truth.
(4) We can look at this example as a case study to evaluate what responses were rational and which weren’t. SoulessAutomaton’s and Anna Salamon’s responses were well-intentioned but escalated the emotional cost of the argument for me (broadly, SoulessAutomaton accused me of being subversive/disrespectful and AnnaSalamon made the character attack that I’m not rational.) Both tempered their meted ‘punishments’ with useful suggestions. VladimirNesov’s comment was I think quite rational: he asserted I probably needed to learn more about the subject and he provided some links. (The specific links are enormously helpful for navigating this huge maze.) One criticism would be that he was overly charitable with his assessment of “affective rhetoric”. While my rhetoric was indeed affective by some measure external to LW, the point is, I know, affective rhetoric for its own sake is not appropriate here. I suspect Vladimir_Nesov was just trying to signal respect for me as an individual before criticizing my position, generally a good practice.
(2) My mistake was not in asserting something false.
It was. What you asserted, depending on interpretation, is either ill-formed or false. A counterexample to your claim is that a Paperclip AI won’t, in any meaningful sense, love humanity.
(3) My transgression was using the emotionally loaded word “love”.
The use of emotionally-loaded word is inappropriate, unless it is. In this case, your statement of attribution of emotion was false, and so affective aura accompanying the statement was inappropriate. I hypothesized that emotional thinking was one of the sources of your belief in the truth of the statement you made, so stating that your words were “affective rhetoric” meant to communicate this diagnostic (by analogy with “empty rhetoric”). I actually edited to that phrase from earlier “affective silliness”, that directly communicated a reference to the fact of you making a mistake, but I changed it to be less offensive.
Vladimir Nesov’s comment was I think quite rational: he asserted I probably needed to learn more about the subject and he provided some links
The ‘probably’ was more of a weasel word, referring to the fact that I’m not sure whether you actually want to spend time learning all that stuff, rather than to special uncertainty in whether the answer to your question is found there.
(1) I’m disappointed that even as rationalists, while you were able to recognize that I had committed some transgression, you were not able to identify it precisely and power was used (authority, shaming) instead of the truth to broadly punish me.
The problem is that the inferential distance is too great, and so it’s easier to refer the newcommer to the archive, where the answer to what was wrong can be learned systematically, instead of trying to explain the problems on her own terms.
I read “affective rhetoric” as “effective rhetoric”. (oops) Yes, “affective rhetoric” is a much more appropriate comment than (“effective rhetoric”). Since it seems like a good place for a neophyte to begin, I will address your comment about the paperclip AI in the welcome thread where Anna Salamon replied.
Anna Salamon replied on the Welcome thread, starting with:
This is in response to a comment of brynema’s elsewhere; if we want LW discussions to thrive even in cases where the discussions require non-trivial prerequisites, my guess is that we should get in the habit of taking “already discussed exhaustively” questions to the welcome thread.
If we want to talk usefully about AI as a community, we should probably make a wiki page that summarizes or links to the main points. And then we should have a policy in certain threads: “don’t comment here unless you’ve read the links off of wiki page such-and-such”.
brynema’s right that we want newcomers in LW, and that newcomers can’t be expected to know all of what’s been discussed. But it is also true that we’ll never get discussions off the ground if we have to start all over again every time someone new enters.
Fieldmice (outside of Douglas Adams fiction) aren’t any particular threat to us in the way we might be to the Unfriendly AI. They’re not likely to program another us to fight us for resources.
You are assuming that mere intelligence is sufficient to give an AI an overwhelming advantage in any conflict. While I concede that is possible in theory I consider it much less likely than seems to be the norm here. This is partly because I am also skeptical about the existential dangers of self replicating nanotech, bioengineered viruses and other such technologies that an AI might attempt to use in a conflict.
As long as there is any reasonable probability that an AI would lose a conflict with humans or suffer serious damage to its capacity to achieve its goals, its best course of action is unlikely to be to attempt to wipe out humanity. A paperclip maximizer for example would seem to better further its goals by heading to the asteroid belt where it could advance its goals without needing to devote large amounts of computational capacity to winning a conflict with other goal-directed agents.
For people who’ve voted this down, I’d be interested in your answers to the following questions:
1) Can you envisage a scenario in which a greater than human intelligence AI with goals not completely compatible with human goals would ever choose a course of action other than wiping out humanity?
2) If you answered yes to 1), what probability do you assign to such an outcome, rather than an outcome involving the complete annihilation of humanity?
3) If you answered no to 1), what makes you certain that such a scenario is not possible?
I agree generally, but I think when we talk about wiping out humanity we should include the idea that if we were to lose a significant portion of our accumulated information it would be essentially the same as extinction. I don’t see a difference between a stone age tech. group of humans surviving the apocalypse and slowly repopulating the world and a different species (whether dogs, squirrels, or porpoises) doing the same thing.
I don’t see a difference between a stone age tech. group of humans surviving the apocalypse and slowly repopulating the world and a different species (whether dogs, squirrels, or porpoises) doing the same thing.
We have pretty solid evidence that a stone age tech group of humans can develop a technologically advanced society in a few 10s of thousands of years. I imagine it would take considerably longer for squirrels to get there and I would be much less confident they can do it at all. It may well be that human intelligence is an evolutionary accident that has only happened once in the universe.
The squirrel civilization would be a pretty impressive achievement, granted. The destruction of this particular species (humans) would seemingly be a tremendous loss universally, if intelligence is a rare thing. Nonetheless, I see it as only a certain vessel in which intelligence happened to arise. I see no particular reason why intelligence should be specific to it, or why we should prefer it over other containers should the opportunity present itself. We would share more in common with an intelligent squirrel civilization than a band of gorillas, even though we would share more genetically with the latter. If I were cryogenically frozen and thawed out a million years later by the world-dominating Squirrel Confederacy, I would certainly live with them rather than seek out my closest primate relatives.
EDIT: I want to expand on this slightly. Say our civilization were to be completely destroyed, and a group of humans that had no contact with us were to develop a new civilization of their own concurrent with a squirrel population doing the same on the other side of the world. If that squirrel civilization were to find some piece of our history, say the design schematics of an electric toothbrush, and adopt it as a part of their knowledge, I would say that for all intents and purposes, the squirrels are more “us” than the humans, and we would survive through the former, not the latter.
I don’t see any fundamental reason why intelligence should be restricted to humans. I think it’s quite possible that intelligence arising in the universe is an extremely rare event though. If you value intelligence and think it might be an unlikely occurrence then the survival of some humans rather than no humans should surely be a much preferred outcome?
I disagree that we would have more in common with the electric toothbrush wielding squirrels. I’ve elaborated more on that in another comment.
Preferred, absolutely. I just think that the survival of our knowledge is more important than the survival of the species sans knowledge. If we are looking to save the world, I think an AI living on the moon pondering its existence should be a higher priority than a hunter-gatherer tribe stalking wildebeest. The former is our heritage, the latter just looks like us.
If I implied that, it was unintentional. All I mean is that I see no reason why we should feel a kinship toward humans as humans, as opposed to any species of people as people. If our civilization were to collapse entirely and had to be rebuilt from scratch, I don’t see why the species that is doing the rebuilding is all that important—they aren’t “us” in any real sense. We can die even if humanity survives. By that same token, if the paperclip AI contains none of our accumulated knowledge, we go extinct along with the species. If the AI contains some our of knowledge and a good degree of sentience, I would argue that part of us survives despite the loss of this particular species.
Bear in mind, the paperclip AI won’t ever look up to the broader challenges of being a sentient being in the Universe; the only thing that will ever matter to it, until the end of time, is paperclips. I wouldn’t feel in that instance that we had left behind a creature that represented our legacy, no matter how much it knows about the Beatles.
OK, I can see that. In that case, maybe a better metric would be the instrumental use of our accumulated knowledge, rather than its mere possession. Living in a library doesn’t mean you can read, after all.
What I think you’re driving at is that you want it to value the Beatles in some way. Having some sort of useful crossover between our values and its is the entire project of FAI.
I’m just trying to figure out under what circumstances we could consider a completely artificial entity a continuation of our existence. As you pointed out, merely containing our knowledge isn’t enough. Human knowledge is a constantly growing edifice, where each generation adds to and build upon the successes of the past. I wouldn’t expect an AI to find value in everything we have produced, just as we don’t. But if our species were wiped out, I would feel comfortable calling an AI which traveled the universe occasionally writing McCartney- or Lennon-inspired songs “us.” That would be survival. (I could even deal with a Ringo Starr AI, in a pinch.)
How much of what it means to be human do you think is cultural conditioning versus innate biological tendency? I think the evidence points to a very large biologically determined element to humanity. I would expect to find more in common with a hunter gatherer in a previously undiscovered tribe, or even with a paleolithic tribesman, than with an alien intelligence or an evolved dolphin.
If you read ancient Greek literature, it is easy to empathize with most of the motivations and drives of the characters even though they lived in a very different world. You could argue that our culture’s direct lineage from theirs is a factor but it seems that westerners can recognize as fellow humans the minds behind ancient Chinese or Indian texts with less shared cultural heritage with our own.
I don’t consider our innate biological tendencies the core of our being. We are an intelligence superimposed on a particular biological creature. It may be difficult to separate the aspects of one from the other (and I don’t pretend to be fully able to do so), but I think it’s important that we learn which is which so that we can slowly deemphasize and discard the biological in favor of the solely rational.
I’m not interested in what it means to be human, I want to know what it means to be a person. Humanity is just an accident as far as I’m concerned. It might as well have been anything else.
I’m curious as to what sorts of goals you think a “solely rational” creature possesses. Do you have a particular point of disagreement with Eliezer’s take on the biological heritage of our values?
Oh, I don’t know that. What would remain of you if you could download your mind into a computer? Who would you be if you were no longer affected by the level of serotonin or adrenaline you are producing, or if pheromones didn’t affect you? Once you subtract the biological from the human, I imagine what remains to be pure person. There should be no difference between that person and one who was created intentionally or one that evolved in a different species, beyond their personal experiences (controlling for the effects of their physiology).
I don’t have any disagreement with Eliezer’s description of how our biology molded our growth, but I see no reason why we should hold on to that biology forever. I could be wrong, however. It may not be possible to be a person without certain biological-like reactions. I can certainly see how this would be the case for people in early learning stages of development, particularly if your goal is to mold that person into a friendly one. Even then, though, I think it would be beneficial to keep those parts to the bare minimum required to function.
What would remain of you if you could download your mind into a computer?
That depends on the resolution of the simulation. Wouldn’t you agree?
Once you subtract the biological from the human, I imagine what remains to be pure person.
I think you’re using the word “biological” to denote some kind of unnatural category.
I don’t have any disagreement with Eliezer’s description of how our biology molded our growth, but I see no reason why we should hold on to that biology forever.
The reasons you see for why any of us “should” do anything almost certainly have biologically engineered goals behind them in some way or another. What of self-preservation?
I meant this kind of unnatural category. I don’t quite know what you mean by “biological” in this context. A high-resolution neurological simulation might not require any physical carbon atoms, but the simulated mind would presumably still act according to all the same “biological” drives.
Pretend that someone says “I’ll give you __ odds, which side do you want?”, and figure out what the odds would have to be to make you indifferent to which side you bet on. Consider the question as if though you were actually going to put money on it .
I will admit to an estimate higher than 95% that humanity or its uploads will survive the next hundred years. Many of the “apocalyptic” scenarios people are concerned about seem unlikely to wipe out all of humanity; so long as we have a breeding population, we can recover.
No significant risk of unFriendly AI (especially since you apparently consider uploading within 100 years plausible)? Nanotech war? Even engineered disease? I’m surprised.
The comment appears to me to be saying there is no significant risk of wiping out all of humanity, not that there is no significant risk of any of the dangers you describe causing significant harm.
I think an unfriendly AI is somewhat likely for example but put a very low probability on an unfriendly AI completely wiping out humanity. The consequences could be quite unpleasant and worth working to avoid but I don’t think it’s an existential threat with any significant probability.
That’s a very strange perspective. Other threats are good in that they are stupid, so they won’t find you if you colonize space or live on an isolated island, or have a lucky combination of genes, or figure out a way to actively outsmart them, etc. Stupid existential risks won’t methodically exterminate every human, and so there is a chance for recovery. Unfriendly AI, on the other hand, won’t go away, and you can’t hide from it on another planet. (Indifference works this way too, it’s the application of power indifferent to humankind that is methodical, e.g. Paperclip AI.)
It’s not very strange. It’s a perspective that tends to match most human intuitions. It is, however, a very wrong perspective.
Consider: humanity is an intelligence, one not particularly friendly to, say, the fieldmouse. Fieldmice are not yet extinct.
I think it is worth considering the number of species to which humanity is largely indifferent which are extinct as a result of humanity optimizing other criteria
Humans satisfice, and not very well at that compared to what an AGI could do. If we effectively optimized for… almost any goal not referring to fieldmice… fieldmice would be extinct.
Humanity is weak.
Humanity is pretty damn impressive from a fieldmouse’s perspective, I dare say!
yet humanity cannot create technology on the level of a fieldmouse.
Humanity is beautiful. A significantly more intelligent AI will love us more perfectly (no: more truly) than we love a field mouse. (It is an intermediately stupid AI that I would worry about.)
Later edit: If you’re interested in reading past group discussion on the topic of how superintelligence does not imply supermorality, search “surface analogies” and “supermorality”.
Affective rhetoric. It seems like you are reasoning by surface analogies that don’t apply, anthropomorphizing AIs without realizing that, thinking that you’ve successfully abstracted away all the human-specific (and irrelevant!) things. Unless you are condensing a deeper model, which you’d need to present in more detail to discuss, you just need to learn more about the subject before drawing any strong conclusions. Read up on the Yudkowsky’s posts. (For this comment in particular: see That Tiny Note of Discord and dependencies, although that’s far from an ideal first post on the subject.)
You assume that an AI will necessarily value beauty as you conceive it. This is unlikely.
Yes, I understand that this is not religion and all positions will have to be argued and defended in due time. I am merely declaring my position. I do find it really fascinating that, in the first stages of drafting this new map, we begin by drawing lines in the sand…
It’s more that the counterargument against your position was covered, at great length, and then covered some more, on OB by Yudkowsky, the person that most of are here because we respect.
If you’re going to take a stand for something that most people here have already read very persuasive arguments against, I don’t think it’s unreasonable to expect more than just a position statement (and an emotionally-loaded one, at that).
I meant no disrespect. (Eliezer has 661 posts on OB.) I do appreciate your direction/correction. I didn’t mean to take a stand against.
(Sigh.) I have no positions, no beliefs, prior to what I might learn from Eliezer.
So the idea is that a unique, complex thing may not necessarily have an appreciation for another unique complexity? Unless appreciating unique complexity has a mathematical basis.
brynema, “disrespect” isn’t at all the the right axis for understanding why your last couple comments weren’t helpful. (I’m not attacking you here; LW is an unusual place, and understanding how to usefully contribute takes time. You’ve been doing well.) The trouble with your last two comments is mostly:
Comments on LW should aspire to rationality. As part of this aspiration, we basically shouldn’t have “positions” on issues we haven’t thought much about; the beliefs we share here should be evidence-based best-guesses about the future, not clothes to decorate ourselves with.
Many places encourage people to make up and share “beliefs”, because any person’s beliefs are as good as any other’s and it’s good to express oneself, or something like that. Those norms are not useful toward arriving at truth, at least not compared to what we usually manage on LW. Not even if people follow their made-up “beliefs” with evidence created to support their conclusions; nor even if evidence or intuitions play some role in the initial forming of beliefs.
This is particularly true in cases where the subjects are difficult technical problems that some in the community have specialized in and thought carefully about; declaring positions there is kind of like approaching a physicist, without knowledge of physics, and announcing your “position” on how atoms hold together. (Though less so, since AI is less well-grounded than physics.)
AI risks are a particularly difficult subject about which to have useful conversation, mostly because there is little data to help keep conversation from veering off into nonsense-land. So it makes sense, in discussing AI risks and other slippery topics, to have lower tolerance for folks making up positions.
Also, yes, these particular positions have been discussed and have proven un-workable in pretty exhaustive detail.
As to the object-level issue concerning possible minds, I wrote an answer in the welcome thread, on the theory that, if we want to talk about AI or other prerequisite-requiring topics on LW, we should probably get in the habit of taking “already discussed to death” questions to the welcome thread, where they won’t clutter mainline discussion. Please don’t be offended by this, though; I value your presence here, and you had no real way of knowing this had already been discussed.
I’ve spent some time working through my emotional responses and intellectual defenses to the posts above. I would like to make some observations:
(1) I’m disappointed that even as rationalists, while you were able to recognize that I had committed some transgression, you were not able to identify it precisely and power was used (authority, shaming) instead of the truth to broadly punish me.
(2) My mistake was not in asserting something false. This happens all the time here and people usually respond more rationally.
(3) My transgression was using the emotionally loaded word “love”. (So SoulessAutomaton actually came close.) The word in this context is taboo for a good reason—I will try to explain but perhaps I will fail: while I believe in love, I should not put the belief in those terms because invoking the word is dark art manipulation; the whole point of rationality is to find a better vocabulary for explaining truth.
(4) We can look at this example as a case study to evaluate what responses were rational and which weren’t. SoulessAutomaton’s and Anna Salamon’s responses were well-intentioned but escalated the emotional cost of the argument for me (broadly, SoulessAutomaton accused me of being subversive/disrespectful and AnnaSalamon made the character attack that I’m not rational.) Both tempered their meted ‘punishments’ with useful suggestions. VladimirNesov’s comment was I think quite rational: he asserted I probably needed to learn more about the subject and he provided some links. (The specific links are enormously helpful for navigating this huge maze.) One criticism would be that he was overly charitable with his assessment of “affective rhetoric”. While my rhetoric was indeed affective by some measure external to LW, the point is, I know, affective rhetoric for its own sake is not appropriate here. I suspect Vladimir_Nesov was just trying to signal respect for me as an individual before criticizing my position, generally a good practice.
It was. What you asserted, depending on interpretation, is either ill-formed or false. A counterexample to your claim is that a Paperclip AI won’t, in any meaningful sense, love humanity.
The use of emotionally-loaded word is inappropriate, unless it is. In this case, your statement of attribution of emotion was false, and so affective aura accompanying the statement was inappropriate. I hypothesized that emotional thinking was one of the sources of your belief in the truth of the statement you made, so stating that your words were “affective rhetoric” meant to communicate this diagnostic (by analogy with “empty rhetoric”). I actually edited to that phrase from earlier “affective silliness”, that directly communicated a reference to the fact of you making a mistake, but I changed it to be less offensive.
The ‘probably’ was more of a weasel word, referring to the fact that I’m not sure whether you actually want to spend time learning all that stuff, rather than to special uncertainty in whether the answer to your question is found there.
The problem is that the inferential distance is too great, and so it’s easier to refer the newcommer to the archive, where the answer to what was wrong can be learned systematically, instead of trying to explain the problems on her own terms.
I read “affective rhetoric” as “effective rhetoric”. (oops) Yes, “affective rhetoric” is a much more appropriate comment than (“effective rhetoric”). Since it seems like a good place for a neophyte to begin, I will address your comment about the paperclip AI in the welcome thread where Anna Salamon replied.
Anna Salamon replied on the Welcome thread, starting with:
If we want to talk usefully about AI as a community, we should probably make a wiki page that summarizes or links to the main points. And then we should have a policy in certain threads: “don’t comment here unless you’ve read the links off of wiki page such-and-such”.
brynema’s right that we want newcomers in LW, and that newcomers can’t be expected to know all of what’s been discussed. But it is also true that we’ll never get discussions off the ground if we have to start all over again every time someone new enters.
Fieldmice (outside of Douglas Adams fiction) aren’t any particular threat to us in the way we might be to the Unfriendly AI. They’re not likely to program another us to fight us for resources.
If fieldmice were in danger of extinction we’d probably move to protect them, not that that would necessarily help them.
Not on another planet, no. But I wonder how practical a constantly accelerating seed ship will turn out to be.
You are assuming that mere intelligence is sufficient to give an AI an overwhelming advantage in any conflict. While I concede that is possible in theory I consider it much less likely than seems to be the norm here. This is partly because I am also skeptical about the existential dangers of self replicating nanotech, bioengineered viruses and other such technologies that an AI might attempt to use in a conflict.
As long as there is any reasonable probability that an AI would lose a conflict with humans or suffer serious damage to its capacity to achieve its goals, its best course of action is unlikely to be to attempt to wipe out humanity. A paperclip maximizer for example would seem to better further its goals by heading to the asteroid belt where it could advance its goals without needing to devote large amounts of computational capacity to winning a conflict with other goal-directed agents.
For people who’ve voted this down, I’d be interested in your answers to the following questions:
1) Can you envisage a scenario in which a greater than human intelligence AI with goals not completely compatible with human goals would ever choose a course of action other than wiping out humanity?
2) If you answered yes to 1), what probability do you assign to such an outcome, rather than an outcome involving the complete annihilation of humanity?
3) If you answered no to 1), what makes you certain that such a scenario is not possible?
I agree generally, but I think when we talk about wiping out humanity we should include the idea that if we were to lose a significant portion of our accumulated information it would be essentially the same as extinction. I don’t see a difference between a stone age tech. group of humans surviving the apocalypse and slowly repopulating the world and a different species (whether dogs, squirrels, or porpoises) doing the same thing.
See In Praise of Boredom and Sympathetic Minds: random evolved intelligent species are not guaranteed to be anything we would consider valuable.
I like humans. I think they’re cute :3
We have pretty solid evidence that a stone age tech group of humans can develop a technologically advanced society in a few 10s of thousands of years. I imagine it would take considerably longer for squirrels to get there and I would be much less confident they can do it at all. It may well be that human intelligence is an evolutionary accident that has only happened once in the universe.
The squirrel civilization would be a pretty impressive achievement, granted. The destruction of this particular species (humans) would seemingly be a tremendous loss universally, if intelligence is a rare thing. Nonetheless, I see it as only a certain vessel in which intelligence happened to arise. I see no particular reason why intelligence should be specific to it, or why we should prefer it over other containers should the opportunity present itself. We would share more in common with an intelligent squirrel civilization than a band of gorillas, even though we would share more genetically with the latter. If I were cryogenically frozen and thawed out a million years later by the world-dominating Squirrel Confederacy, I would certainly live with them rather than seek out my closest primate relatives.
EDIT: I want to expand on this slightly. Say our civilization were to be completely destroyed, and a group of humans that had no contact with us were to develop a new civilization of their own concurrent with a squirrel population doing the same on the other side of the world. If that squirrel civilization were to find some piece of our history, say the design schematics of an electric toothbrush, and adopt it as a part of their knowledge, I would say that for all intents and purposes, the squirrels are more “us” than the humans, and we would survive through the former, not the latter.
I don’t see any fundamental reason why intelligence should be restricted to humans. I think it’s quite possible that intelligence arising in the universe is an extremely rare event though. If you value intelligence and think it might be an unlikely occurrence then the survival of some humans rather than no humans should surely be a much preferred outcome?
I disagree that we would have more in common with the electric toothbrush wielding squirrels. I’ve elaborated more on that in another comment.
Preferred, absolutely. I just think that the survival of our knowledge is more important than the survival of the species sans knowledge. If we are looking to save the world, I think an AI living on the moon pondering its existence should be a higher priority than a hunter-gatherer tribe stalking wildebeest. The former is our heritage, the latter just looks like us.
Does this imply that you are OK with a Paperclip AI wiping out humanity, since it will be an intelligent life form much more developed than we are?
If I implied that, it was unintentional. All I mean is that I see no reason why we should feel a kinship toward humans as humans, as opposed to any species of people as people. If our civilization were to collapse entirely and had to be rebuilt from scratch, I don’t see why the species that is doing the rebuilding is all that important—they aren’t “us” in any real sense. We can die even if humanity survives. By that same token, if the paperclip AI contains none of our accumulated knowledge, we go extinct along with the species. If the AI contains some our of knowledge and a good degree of sentience, I would argue that part of us survives despite the loss of this particular species.
Bear in mind, the paperclip AI won’t ever look up to the broader challenges of being a sentient being in the Universe; the only thing that will ever matter to it, until the end of time, is paperclips. I wouldn’t feel in that instance that we had left behind a creature that represented our legacy, no matter how much it knows about the Beatles.
OK, I can see that. In that case, maybe a better metric would be the instrumental use of our accumulated knowledge, rather than its mere possession. Living in a library doesn’t mean you can read, after all.
What I think you’re driving at is that you want it to value the Beatles in some way. Having some sort of useful crossover between our values and its is the entire project of FAI.
I’m just trying to figure out under what circumstances we could consider a completely artificial entity a continuation of our existence. As you pointed out, merely containing our knowledge isn’t enough. Human knowledge is a constantly growing edifice, where each generation adds to and build upon the successes of the past. I wouldn’t expect an AI to find value in everything we have produced, just as we don’t. But if our species were wiped out, I would feel comfortable calling an AI which traveled the universe occasionally writing McCartney- or Lennon-inspired songs “us.” That would be survival. (I could even deal with a Ringo Starr AI, in a pinch.)
I strongly suspect that that is the same thing as a Friendly AI, and therefore I still consider UFAI an existential risk.
The Paperclip AI will optimally use its knowledge about the Beatles to make more paperclips.
How much of what it means to be human do you think is cultural conditioning versus innate biological tendency? I think the evidence points to a very large biologically determined element to humanity. I would expect to find more in common with a hunter gatherer in a previously undiscovered tribe, or even with a paleolithic tribesman, than with an alien intelligence or an evolved dolphin.
If you read ancient Greek literature, it is easy to empathize with most of the motivations and drives of the characters even though they lived in a very different world. You could argue that our culture’s direct lineage from theirs is a factor but it seems that westerners can recognize as fellow humans the minds behind ancient Chinese or Indian texts with less shared cultural heritage with our own.
I don’t consider our innate biological tendencies the core of our being. We are an intelligence superimposed on a particular biological creature. It may be difficult to separate the aspects of one from the other (and I don’t pretend to be fully able to do so), but I think it’s important that we learn which is which so that we can slowly deemphasize and discard the biological in favor of the solely rational.
I’m not interested in what it means to be human, I want to know what it means to be a person. Humanity is just an accident as far as I’m concerned. It might as well have been anything else.
I’m curious as to what sorts of goals you think a “solely rational” creature possesses. Do you have a particular point of disagreement with Eliezer’s take on the biological heritage of our values?
Oh, I don’t know that. What would remain of you if you could download your mind into a computer? Who would you be if you were no longer affected by the level of serotonin or adrenaline you are producing, or if pheromones didn’t affect you? Once you subtract the biological from the human, I imagine what remains to be pure person. There should be no difference between that person and one who was created intentionally or one that evolved in a different species, beyond their personal experiences (controlling for the effects of their physiology).
I don’t have any disagreement with Eliezer’s description of how our biology molded our growth, but I see no reason why we should hold on to that biology forever. I could be wrong, however. It may not be possible to be a person without certain biological-like reactions. I can certainly see how this would be the case for people in early learning stages of development, particularly if your goal is to mold that person into a friendly one. Even then, though, I think it would be beneficial to keep those parts to the bare minimum required to function.
That depends on the resolution of the simulation. Wouldn’t you agree?
I think you’re using the word “biological” to denote some kind of unnatural category.
The reasons you see for why any of us “should” do anything almost certainly have biologically engineered goals behind them in some way or another. What of self-preservation?
Not unnatural, obviously, but a contaminant to intelligence. Manure is a great fertilizer, but you wash it off before you use the vegetable.
I meant this kind of unnatural category. I don’t quite know what you mean by “biological” in this context. A high-resolution neurological simulation might not require any physical carbon atoms, but the simulated mind would presumably still act according to all the same “biological” drives.
I’m certain.
I take much the same position.