As I understand it, EY worked through a chain of reasoning about a decade ago, in his book “Creating Friendly AI”. The chain of reasoning is long and I won’t attempt to recap it here, but there are two relevant conclusions.
First, that self-improving artificial intelligences are dangerous, and that projects to build self-improving artificial intelligence, or general intelligence that might in principle become self-modifying (such as Goertzel’s), are increasing existential risk.
Second, that the primary defense against self-improving artificial intelligences is a Friendly self-improving artificial intelligence, and so, in order to reduce existential risk, EY must work on developing (a restricted subset of) self-improving artificial intelligence.
This seems nigh-paradoxical (and unnecessarily dramatic) to me—you should not do , and yet EY must do . As I said before, this “cancel infinities against one another” sort of thinking (another example might be MAD doctrine), has enormous appeal to a certain (geeky) kind of person. The phenomenon is named “nerd-sniping” in the xkcd comic: http://xkcd.com/356/
Rather than pursuing Friendly AGI vigorously as last/best/only hope for humanity, we should do at least two things:
Look hard for errors in the long chain of reasoning that led to these peculiar conclusions, on the grounds that reality rarely calls for that kind of nigh-paradoxical action, and it’s far more likely that either all AI development is generally a good thing for existential risks, or all AI development is a generally bad thing for existential risks—EY shouldn’t get any special AI-development license.
Look hard for more choices—for example, building entities that are very capable at defeating rogue Unfriendly AGI takeoffs, and yet which are not themselves a threat to humanity in general, nor prone to hard takeoffs. It may be difficult to imagine such entities, but all the reduce-existential-risk tasks are very difficult.
reality rarely calls for that kind of nigh-paradoxical action
In my experience, reality frequently includes scenarios where the best way to improve my ability to defend myself involves also improving my ability to harm others, should I decide to do that. So it doesn’t seem that implausible to me.
Indeed, militaries are pretty much built on this principle, and are fairly common.
I am familiar with the libertarian argument that if everyone has more destructive power, the society is safer. The analogous position would be that if everyone pursues (Friendly) AGI vigorously, existential risk would be reduced. That might well be reasonable, but as far as I can tell, that’s NOT what is advocated.
Rather, we are all asked to avoid AGI research (and go into software development and make money and donate? How much safer is general software development for a corporation than careful AGI research?) and instead sponsor SIAI/EY doing (Friendly) AGI research while SIAI/EY is fairly closed-mouth about it.
It just seems to me like it would take a terribly delicate balance of probabilities to make this the safest course forward.
I have similar misgivings, they prompted me to write the post. Fighting fire with fire looks like a dangerous idea. The problem statement should look like “how do we stop unfriendly AIs”, not “how do we make friendly AIs”. Many people here (e.g. Nesov and SarahC) seem convinced that the latter is the most efficient way of achieving the former. I hope we can find a better way if we think some more.
The problem statement should look like “how do we stop unfriendly AIs”, not “how do we make friendly AIs”.
If the universe is capable of running super-intelligent beings, then eventually either there will be one, or civilization will collapse. Maintaining the current state where there are no minds more intelligent than base humans seems very unlikely to be stable in the long run.
Given that, it seems the problem should be framed as “how do we end up with a super-intelligent being (or beings) that will go on to rearrange the universe the way we prefer?” which is not too different from “how do we make friendly AIs” if we interpret things like recursively-improved uploads as AIs.
As I understand it, EY worked through a chain of reasoning about a decade ago, in his book “Creating Friendly AI”. The chain of reasoning is long and I won’t attempt to recap it here, but there are two relevant conclusions.
First, that self-improving artificial intelligences are dangerous, and that projects to build self-improving artificial intelligence, or general intelligence that might in principle become self-modifying (such as Goertzel’s), are increasing existential risk. Second, that the primary defense against self-improving artificial intelligences is a Friendly self-improving artificial intelligence, and so, in order to reduce existential risk, EY must work on developing (a restricted subset of) self-improving artificial intelligence.
This seems nigh-paradoxical (and unnecessarily dramatic) to me—you should not do , and yet EY must do . As I said before, this “cancel infinities against one another” sort of thinking (another example might be MAD doctrine), has enormous appeal to a certain (geeky) kind of person. The phenomenon is named “nerd-sniping” in the xkcd comic: http://xkcd.com/356/
Rather than pursuing Friendly AGI vigorously as last/best/only hope for humanity, we should do at least two things:
Look hard for errors in the long chain of reasoning that led to these peculiar conclusions, on the grounds that reality rarely calls for that kind of nigh-paradoxical action, and it’s far more likely that either all AI development is generally a good thing for existential risks, or all AI development is a generally bad thing for existential risks—EY shouldn’t get any special AI-development license.
Look hard for more choices—for example, building entities that are very capable at defeating rogue Unfriendly AGI takeoffs, and yet which are not themselves a threat to humanity in general, nor prone to hard takeoffs. It may be difficult to imagine such entities, but all the reduce-existential-risk tasks are very difficult.
In my experience, reality frequently includes scenarios where the best way to improve my ability to defend myself involves also improving my ability to harm others, should I decide to do that. So it doesn’t seem that implausible to me.
Indeed, militaries are pretty much built on this principle, and are fairly common.
But, sure… there are certainly alternatives.
I am familiar with the libertarian argument that if everyone has more destructive power, the society is safer. The analogous position would be that if everyone pursues (Friendly) AGI vigorously, existential risk would be reduced. That might well be reasonable, but as far as I can tell, that’s NOT what is advocated.
Rather, we are all asked to avoid AGI research (and go into software development and make money and donate? How much safer is general software development for a corporation than careful AGI research?) and instead sponsor SIAI/EY doing (Friendly) AGI research while SIAI/EY is fairly closed-mouth about it.
It just seems to me like it would take a terribly delicate balance of probabilities to make this the safest course forward.
I have similar misgivings, they prompted me to write the post. Fighting fire with fire looks like a dangerous idea. The problem statement should look like “how do we stop unfriendly AIs”, not “how do we make friendly AIs”. Many people here (e.g. Nesov and SarahC) seem convinced that the latter is the most efficient way of achieving the former. I hope we can find a better way if we think some more.
If the universe is capable of running super-intelligent beings, then eventually either there will be one, or civilization will collapse. Maintaining the current state where there are no minds more intelligent than base humans seems very unlikely to be stable in the long run.
Given that, it seems the problem should be framed as “how do we end up with a super-intelligent being (or beings) that will go on to rearrange the universe the way we prefer?” which is not too different from “how do we make friendly AIs” if we interpret things like recursively-improved uploads as AIs.