I agree that a write-up of SIAI’s argument for the Scary Idea, in the manner you describe, would be quite interesting to see.
However, I strongly suspect that when the argument is laid out formally, what we’ll find is that
-- given our current knowledge about the pdf’s of the premises in the argument, the pdf on the conclusion is verrrrrrry broad, i.e. we can’t conclude hardly anything with much of any confidence …
So, I think that the formalization will lead to the conclusion that
-- “we can NOT confidently say, now, that: Building advanced AGI without a provably Friendly design will almost certainly lead to bad consequences for humanity”
-- “we can also NOT confidently say, now, that: Building advanced AGI without a provably Friendly design will almost certainly NOT lead to bad consequences for humanity”
I.e., I strongly suspect the formalization
-- will NOT support the Scary Idea
-- will also not support complacency about AGI safety and AGI existential risk
I think the conclusion of the formalization exercise, if it’s conducted, will basically be to reaffirm common sense, rather than to bolster extreme views like the Scary Idea....
So, I think that the formalization will lead to the conclusion that
“we can NOT confidently say, now, that: Building advanced AGI without a provably Friendly design will almost certainly lead to bad consequences for humanity”
“we can also NOT confidently say, now, that: Building advanced AGI without a provably Friendly design will almost certainly NOT lead to bad consequences for humanity”
I agree with both those statements, but think the more relevant question would be:
“conditional on it turning out, to the enormous surprise of most everyone in AI, that this AGI design is actually very close to producing an ‘artificial toddler’, what is the sign of the expected effect on the probability of an OK outcome for the world, long-term and taking into account both benefits and risks?”
.
I doubt you would remember this, but we talked about this at the Meet and Greet at the Singularity Summit a few months ago (in addition to CBGBs and Punk Rock and Skaters).
James Hughes mentioned you as well at a Conference in NY where we discussed this very issue as well.
One thing that you mentioned at the Summit (well in conversation) was that The Scary Idea was tending to cause some paranoia among people who otherwise might be contributing more to the development of AI (of course, you also seemed pretty hostile to brain emulation too) as it tends to cause funding that could be going to AI to be slowed as a result.
I, personally, think that you’ve misunderstood something. It’s not that designing a non-provably Friendly AI is almost certain to destroy humanity, it’s that it’s possible and it’s good engineering practice is to assume the worst is going to happen and design based on that.
Define “sufficiently low”; with even a 99.9% chance of success, you’ve still got a .1% chance of killing every human alive; that’s morally equivalent to a 100% chance of killing 6.5 million people. Saying that if you’re not totally sure that if you’re not totally sure that your AI is Friendly when you start it up, you’re committing the Holocaust was not hyperbole in any way, shape, or form. It’s simply the result of shutting up and doing the multiplication.
If you calculate a .1% chance of killing every human alive if you start it right now, but also a .2% chance of saving the whole of humanity, that’s morally equivalent to a 100% chance of saving the lives of 6.5 million people—in which case you’re guilty of the Holocaust if you do NOT start it.
Negligible in terms of calculating the effect of the AI project on existential risk, because the other effects, positive and negative, would be so much larger.
If you’re reducing the expected risk of existential disaster by a larger amount, you’re in expectation net saving lives rather than net killing. If all options involve existential risk, including doing nothing, then all one can do is pick the option with the lowest.
I agree that a write-up of SIAI’s argument for the Scary Idea, in the manner you describe, would be quite interesting to see.
However, I strongly suspect that when the argument is laid out formally, what we’ll find is that
-- given our current knowledge about the pdf’s of the premises in the argument, the pdf on the conclusion is verrrrrrry broad, i.e. we can’t conclude hardly anything with much of any confidence …
So, I think that the formalization will lead to the conclusion that
-- “we can NOT confidently say, now, that: Building advanced AGI without a provably Friendly design will almost certainly lead to bad consequences for humanity”
-- “we can also NOT confidently say, now, that: Building advanced AGI without a provably Friendly design will almost certainly NOT lead to bad consequences for humanity”
I.e., I strongly suspect the formalization
-- will NOT support the Scary Idea
-- will also not support complacency about AGI safety and AGI existential risk
I think the conclusion of the formalization exercise, if it’s conducted, will basically be to reaffirm common sense, rather than to bolster extreme views like the Scary Idea....
-- Ben Goertzel
I agree with both those statements, but think the more relevant question would be:
“conditional on it turning out, to the enormous surprise of most everyone in AI, that this AGI design is actually very close to producing an ‘artificial toddler’, what is the sign of the expected effect on the probability of an OK outcome for the world, long-term and taking into account both benefits and risks?” .
[deleted]
I agree.
I doubt you would remember this, but we talked about this at the Meet and Greet at the Singularity Summit a few months ago (in addition to CBGBs and Punk Rock and Skaters).
James Hughes mentioned you as well at a Conference in NY where we discussed this very issue as well.
One thing that you mentioned at the Summit (well in conversation) was that The Scary Idea was tending to cause some paranoia among people who otherwise might be contributing more to the development of AI (of course, you also seemed pretty hostile to brain emulation too) as it tends to cause funding that could be going to AI to be slowed as a result.
I, personally, think that you’ve misunderstood something. It’s not that designing a non-provably Friendly AI is almost certain to destroy humanity, it’s that it’s possible and it’s good engineering practice is to assume the worst is going to happen and design based on that.
A sufficiently low probability becomes negligible in light of other risks and risk-reduction opportunities.
Define “sufficiently low”; with even a 99.9% chance of success, you’ve still got a .1% chance of killing every human alive; that’s morally equivalent to a 100% chance of killing 6.5 million people. Saying that if you’re not totally sure that if you’re not totally sure that your AI is Friendly when you start it up, you’re committing the Holocaust was not hyperbole in any way, shape, or form. It’s simply the result of shutting up and doing the multiplication.
If you calculate a .1% chance of killing every human alive if you start it right now, but also a .2% chance of saving the whole of humanity, that’s morally equivalent to a 100% chance of saving the lives of 6.5 million people—in which case you’re guilty of the Holocaust if you do NOT start it.
“Shut up and multiply” works both ways.
This doesn’t hold if some extra work could improve those odds.
(IMO, the sense of moral urgency created by things like Holocaust analogies almost always does more harm than good.)
Negligible in terms of calculating the effect of the AI project on existential risk, because the other effects, positive and negative, would be so much larger.
Any other possible effects don’t negate that you’re killing six million people when you’re going ahead with a potentially UnFriendly AI.
If you’re reducing the expected risk of existential disaster by a larger amount, you’re in expectation net saving lives rather than net killing. If all options involve existential risk, including doing nothing, then all one can do is pick the option with the lowest.