I don’t think you actually hold this view. It is logically inconsistent with practices like eating food.
It might not be. He has certain short term goals of the form “while I’m alive, I’d like to do X” that’s very different from goals connected to the general success of humanity.
Ooops, logically inconsistent was way too strong. I got carried away with making a point. I was reasoning that: “eat food” is a evolutionary drive; “produce descendants that survive” is also an evolutionary drive; “a human future” wholly contains futures where his descendants survive. From that I concluded that it is unlikely he has no evolutionary drives—I didn’t consider the possibility that he is missing some evolutionary drives, including all ones that require a human future—and therefore he is tied to a human future, but finds it expedient for other reasons (contrarian signaling, not admitting defeat in an argument) to claim he doesn’t.
It’s a finite loss (6.8x10^9 multiplied by loss of 1 human life) but I definitely understand why it looks infinite:
I should have been more clear: I mean, if we believe in the scary idea, there are two effects:
Some set of grandmas die. (finite, comparatively small loss)
Humanity is more likely to go extinct due to an unfriendly AGI. (infinite, comparatively large loss; infinite because of the future humans that would have existed but don’t.)
Now, the benefit of believing the Scary Idea is that humanity is less likely to go extinct due to an unfriendly AGI- but my point is that you are not wagering on separate scales (low chance of infinite gain? Sign me up!) but that you are wagering on the same scale (an unfriendly AGI appears!), and the effects of your wager are unknown.
“produce descendants that survive” is also an evolutionary drive
And who said anything about those descendants having to be human?
This answers your other question: yes, I would be willing to have children normally, I would be willing to kill to protect my children, and I would be willing to die to protect my children.
The best-case scenario is that we can have those children and they respect (though they surpass) their parents- the worst-case scenario is we die in childbirth. But all of those are things I can be comfortable with.
(I will note that I’m assuming here the AGI surpasses us. It’s not clear to me that a paperclip-maker does, but it is clear to me that there can be an AGI who is unfriendly solely because we are inconvenient and does surpass us. So I would try and make sure it doesn’t just focus on making paperclips, but wouldn’t focus too hard on making sure it wants me to stick around.)
The best-case scenario is that we can have those children and they respect (though they surpass) their parents- the worst-case scenario is we die in childbirth. But all of those are things I can be comfortable with.
Well, the worst case scenario is that you die in childbirth and take the entire human race with you. That is not something I am comfortable with, regardless of whether you are. And you said you are willing to kill to protect your children. You think some of the Scary Idea proponents could be parents with children, and they don’t want to see their kids die because you gave birth to an AI?
Well, the worst case scenario is that you die in childbirth and take the entire human race with you. That is not something I am comfortable with, regardless of whether you are. And you said you are willing to kill to protect your children. You think some of the Scary Idea proponents could be parents with children, and they don’t want to see their kids die because you gave birth to an AI?
I suspect we are at most one more iteration from mutual understanding; we certainly are rapidly approaching it.
If you believe that an AGI will FOOM, then all that matters is the first AGI made. There is no prize for second place. A belief in the Scary Idea has two effects: it makes your AGI more likely to be friendly (since you’re more careful!) and it makes the AGI less likely to be your AGI (since you’re more careful).
Now, one can hope that the Scary Idea meme’s second effect won’t matter, because the meme is so infectious- all you need to do is infect every AI researcher in the world, and now everyone will be more careful and no one will have a carefulness speed disadvantage. But there are two bits of evidence that make that a poor strategy: AI researchers who are familiar with the argument and don’t buy it, and people who buy the argument, but plan to use it to your disadvantage (since now they’re more likely to define the future than you are!).
The scary idea as a technical argument is weighted on unknown and unpredictable values, and the underlying moral argument (to convince someone they should adopt this reasoning) requires that they believe they should weight the satisfaction of other humans more than their ability to define the future, which is a hard sell.
Thus, my statement is, if you care about your children / your ability to define the future / maximizing the likelihood of a friendly AGI / your personal well-being, then believing in the Scary Idea seems counterproductive.
Ok, holy crap. I am going to call this the Really Scary Idea. I had not thought there could be people out there who would actually value being first with the AGI over decreasing the risk of existential disaster, but it is entirely plausible. Thank you for highlighting this for me, I really am grateful. If a little concerned.
Mind projection fallacy, perhaps? I thought the human race was more important than being the guy who invented AGI, so everyone naturally thinks that?
To reply to my own quote, then:
Well, the worst case scenario is that you die in childbirth and take the entire human race with you. That is not something I am comfortable with, regardless of whether you are.
It doesn’t matter what you are comfortable with, if the developer doesn’t have a term in their utility function for your comfort level. Even I have thought similar thoughts with regards to Luddites and such; drag them kicking and screaming into the future if we have to, etc.
I think the best way to think about it, since it helps keep the scope manageable and crystallize the relevant factors, is that it’s not “being first with the AGI” but “defining the future” (the first is the instrumental value, the second is the terminal value). That’s essentially what all existential risk management is about- defining the future, hopefully to not include the vanishing of us / our descendants.
But how you want to define the future- i.e. the most political terminal value you can have- is not written on the universe. So the mind projection fallacy does seem to apply.
The thing that I find odd, though I can’t find the source at the moment (I thought it was Goertzel’s article, but I didn’t find it by a quick skim; it may be in the comments somewhere), is that the SIAI seems to have had the Really Scary Idea first (we want Friendly AI, so we want to be the first to make it, since we can’t trust other people) and then progressed to the Scary Idea (hmm, we can’t trust ourselves to make a Friendly AI). I wonder if the originators of the Scary Idea forgot the Really Scary Idea or never feared it in the first place?
Making a superintelligence you don’t want before you make the superintelligence you do want, has the same consequences as someone else building a superintelligence you don’t want before you build the superintelligence you do want.
You might argue that you could make a less bad superintelligence that you don’t want than someone else, but we don’t care very much about the difference between tiling the universe with paperclips and tiling the universe with molecular smiley faces.
I’m sorry, but I extracted no novel information from this reply. I’m aware that FAI is a non-trivial problem, and I think work done on making AI more likely to be FAI has value.
But that doesn’t mean believing the Scary Idea, or discussing the Scary Idea without also discussing the Really Scary Idea, decreases the existential risk involved. The estimations involved have almost no dependence on evidence, and so it’s just comparison of priors, which does not seem sufficient to make a strong recommendation.
It may help if you view my objections as pointing out that the Scary Idea is privileging a hypothesis, not that the Scary Idea is something we should ignore.
No. Expecting a superintelligence to optimize for our specific values would be privileging a hypothesis. The “Scary Idea” is saying that most likely something else will happen.
I may have to start only writing thousand-word replies, in the hopes that I can communicate more clearly in such a format.
There are two aspects to the issue of how much work should be put into FAI as I understand it. The first I word like this- “the more thought we put into whether or not an AGI will be friendly, the more likely the AGI will be friendly.” The second I word like this- “the more thought we put into making our AGI, the less likely our AGI will be the AGI.” Both are wrapped up in the Scary Idea- the first part is it as normally stated, the second part is its unstated consequence. The value of believing the Scary Idea is the benefit of the first minus the cost of the second.
My understanding is that we have no good estimation of the value of the first aspect or the second aspect. This isn’t astronomy where we have a good idea of the number of asteroids out there and a pretty good idea of how they move through space. And so, to declare that the first aspect is stronger without evidence strikes me as related to privileging the hypothesis.
(I should note that I expect, without evidence, the problem of FAI to be simpler than the problem of AGI, and thus don’t think the Scary Idea has any policy implications besides “someone should work on FAI.” The risk that AGI gets solved before FAI means more people should work on FAI, not that less people should work on AGI.)
Expecting a superintelligence to optimize for our specific values would be privileging a hypothesis. The “Scary Idea” is saying that most likely something else will happen.
That is not exactly what Goertzel meant by “Scary Idea”. He wrote:
Roughly, the Scary Idea posits that: If I or anybody else actively trying to build advanced AGI succeeds, we’re highly likely to cause an involuntary end to the human race.
It seems to me that there may be a lot of wiggle room in between failing to “optimize for our specific values” and causing “an involuntary end to the human race”. The human race is not so automatically so fragile that it can only survive under the care of a god constructed in our own image.
It might not be. He has certain short term goals of the form “while I’m alive, I’d like to do X” that’s very different from goals connected to the general success of humanity.
Ooops, logically inconsistent was way too strong. I got carried away with making a point. I was reasoning that: “eat food” is a evolutionary drive; “produce descendants that survive” is also an evolutionary drive; “a human future” wholly contains futures where his descendants survive. From that I concluded that it is unlikely he has no evolutionary drives—I didn’t consider the possibility that he is missing some evolutionary drives, including all ones that require a human future—and therefore he is tied to a human future, but finds it expedient for other reasons (contrarian signaling, not admitting defeat in an argument) to claim he doesn’t.
I should have been more clear: I mean, if we believe in the scary idea, there are two effects:
Some set of grandmas die. (finite, comparatively small loss)
Humanity is more likely to go extinct due to an unfriendly AGI. (infinite, comparatively large loss; infinite because of the future humans that would have existed but don’t.)
Now, the benefit of believing the Scary Idea is that humanity is less likely to go extinct due to an unfriendly AGI- but my point is that you are not wagering on separate scales (low chance of infinite gain? Sign me up!) but that you are wagering on the same scale (an unfriendly AGI appears!), and the effects of your wager are unknown.
And who said anything about those descendants having to be human?
This answers your other question: yes, I would be willing to have children normally, I would be willing to kill to protect my children, and I would be willing to die to protect my children.
The best-case scenario is that we can have those children and they respect (though they surpass) their parents- the worst-case scenario is we die in childbirth. But all of those are things I can be comfortable with.
(I will note that I’m assuming here the AGI surpasses us. It’s not clear to me that a paperclip-maker does, but it is clear to me that there can be an AGI who is unfriendly solely because we are inconvenient and does surpass us. So I would try and make sure it doesn’t just focus on making paperclips, but wouldn’t focus too hard on making sure it wants me to stick around.)
Well, the worst case scenario is that you die in childbirth and take the entire human race with you. That is not something I am comfortable with, regardless of whether you are. And you said you are willing to kill to protect your children. You think some of the Scary Idea proponents could be parents with children, and they don’t want to see their kids die because you gave birth to an AI?
I suspect we are at most one more iteration from mutual understanding; we certainly are rapidly approaching it.
If you believe that an AGI will FOOM, then all that matters is the first AGI made. There is no prize for second place. A belief in the Scary Idea has two effects: it makes your AGI more likely to be friendly (since you’re more careful!) and it makes the AGI less likely to be your AGI (since you’re more careful).
Now, one can hope that the Scary Idea meme’s second effect won’t matter, because the meme is so infectious- all you need to do is infect every AI researcher in the world, and now everyone will be more careful and no one will have a carefulness speed disadvantage. But there are two bits of evidence that make that a poor strategy: AI researchers who are familiar with the argument and don’t buy it, and people who buy the argument, but plan to use it to your disadvantage (since now they’re more likely to define the future than you are!).
The scary idea as a technical argument is weighted on unknown and unpredictable values, and the underlying moral argument (to convince someone they should adopt this reasoning) requires that they believe they should weight the satisfaction of other humans more than their ability to define the future, which is a hard sell.
Thus, my statement is, if you care about your children / your ability to define the future / maximizing the likelihood of a friendly AGI / your personal well-being, then believing in the Scary Idea seems counterproductive.
Ok, holy crap. I am going to call this the Really Scary Idea. I had not thought there could be people out there who would actually value being first with the AGI over decreasing the risk of existential disaster, but it is entirely plausible. Thank you for highlighting this for me, I really am grateful. If a little concerned.
Mind projection fallacy, perhaps? I thought the human race was more important than being the guy who invented AGI, so everyone naturally thinks that?
To reply to my own quote, then:
It doesn’t matter what you are comfortable with, if the developer doesn’t have a term in their utility function for your comfort level. Even I have thought similar thoughts with regards to Luddites and such; drag them kicking and screaming into the future if we have to, etc.
And… mutual understanding in one!
I think the best way to think about it, since it helps keep the scope manageable and crystallize the relevant factors, is that it’s not “being first with the AGI” but “defining the future” (the first is the instrumental value, the second is the terminal value). That’s essentially what all existential risk management is about- defining the future, hopefully to not include the vanishing of us / our descendants.
But how you want to define the future- i.e. the most political terminal value you can have- is not written on the universe. So the mind projection fallacy does seem to apply.
The thing that I find odd, though I can’t find the source at the moment (I thought it was Goertzel’s article, but I didn’t find it by a quick skim; it may be in the comments somewhere), is that the SIAI seems to have had the Really Scary Idea first (we want Friendly AI, so we want to be the first to make it, since we can’t trust other people) and then progressed to the Scary Idea (hmm, we can’t trust ourselves to make a Friendly AI). I wonder if the originators of the Scary Idea forgot the Really Scary Idea or never feared it in the first place?
Making a superintelligence you don’t want before you make the superintelligence you do want, has the same consequences as someone else building a superintelligence you don’t want before you build the superintelligence you do want.
You might argue that you could make a less bad superintelligence that you don’t want than someone else, but we don’t care very much about the difference between tiling the universe with paperclips and tiling the universe with molecular smiley faces.
I’m sorry, but I extracted no novel information from this reply. I’m aware that FAI is a non-trivial problem, and I think work done on making AI more likely to be FAI has value.
But that doesn’t mean believing the Scary Idea, or discussing the Scary Idea without also discussing the Really Scary Idea, decreases the existential risk involved. The estimations involved have almost no dependence on evidence, and so it’s just comparison of priors, which does not seem sufficient to make a strong recommendation.
It may help if you view my objections as pointing out that the Scary Idea is privileging a hypothesis, not that the Scary Idea is something we should ignore.
No. Expecting a superintelligence to optimize for our specific values would be privileging a hypothesis. The “Scary Idea” is saying that most likely something else will happen.
I may have to start only writing thousand-word replies, in the hopes that I can communicate more clearly in such a format.
There are two aspects to the issue of how much work should be put into FAI as I understand it. The first I word like this- “the more thought we put into whether or not an AGI will be friendly, the more likely the AGI will be friendly.” The second I word like this- “the more thought we put into making our AGI, the less likely our AGI will be the AGI.” Both are wrapped up in the Scary Idea- the first part is it as normally stated, the second part is its unstated consequence. The value of believing the Scary Idea is the benefit of the first minus the cost of the second.
My understanding is that we have no good estimation of the value of the first aspect or the second aspect. This isn’t astronomy where we have a good idea of the number of asteroids out there and a pretty good idea of how they move through space. And so, to declare that the first aspect is stronger without evidence strikes me as related to privileging the hypothesis.
(I should note that I expect, without evidence, the problem of FAI to be simpler than the problem of AGI, and thus don’t think the Scary Idea has any policy implications besides “someone should work on FAI.” The risk that AGI gets solved before FAI means more people should work on FAI, not that less people should work on AGI.)
That is not exactly what Goertzel meant by “Scary Idea”. He wrote:
It seems to me that there may be a lot of wiggle room in between failing to “optimize for our specific values” and causing “an involuntary end to the human race”. The human race is not so automatically so fragile that it can only survive under the care of a god constructed in our own image.
Yes, what I described was not what Goertzel called the “Scary Idea”, but, in context, it describes the aspect of it that we were discussing.