Could the NSA, the security agency of the most powerful country on Earth, implement any of these schemes?
Er, yes, very easily.
Do you believe that if Obama were to ask the NSA to take over Russia, that the NSA could easily do so? If so, I am speechless.
Let’s look at one of the most realistic schemes, creating a bioweapon. Yes, an organization like the NSA could probably design such a bioweapon. But how exactly could they take over the world that way?
They could either use the bioweapon to kill a huge number of people, or use it to blackmail the world into submission. I believe that the former would cause our technological civilization, on which the NSA depends, to collapse. So that would be stupid. The latter would maybe work for some time, until the rest of the world got together, in order to make a believable threat of mutual destruction.
I just don’t see this to be a viable way to take over the world. At least not in such a way that you would gain actual control.
Now I can of course imagine a different world, in which it would be possible to gain control. Such as a world in which everyone important was using advanced brain implants. If these brain implants could be hacked, even the NSA could take over the world. That’s a no-brainer.
I can also imagine a long-term plan. But those are very risky. The longer it takes, the higher the chance that your plan is revealed. Also, other AI’s, with different, opposing utility-functions, will be employed. Some will be used to detect such plans.
Anyway, the assumption that an AI could understand human motivation, and become a skilled manipulator, is already too far-fetched for me to take seriously. People around here too often confound theory with practice. That all this might be physically possible does not prove that it is at all likely.
Do you believe that if Obama were to ask the NSA to take over Russia, that the NSA could easily do so?
No. I think the phrase “take over” is describing two very different scenarios if we compare “Obama trying to take over the world” and “a hypothetical hostile AI trying to take over the world”. Obama has many human scruples and cares a lot about continued human survival, and specifically not just about the continued existence of the people of the USA but that they thrive. (Thankfully!)
I entirely agree that killing huge numbers of people would be a stupid thing for the actual NSA and/or Obama to do. Killing all the people, themselves included, would not only fail to achieve any of their goals but thwart (almost) all of them permanently. I was treating it as part of the premises of the discussion that the AI is at least indifferent to doing so: it needs only enough infrastructure left for it to continue to exist and be able to rebuild under its own total control.
a long-term plan. But those are very risky. The longer it takes, the higher the chance that your plan is revealed.
Yes, indeed, the longer it takes the higher the chance that the plan is revealed. But a different plan may take longer but still have a lower overall chance of failure if its risk of discovery per unit time is substantially lower. Depending on the circumstances, one can imagine an AI calculating that its best interests lie in a plan that takes a very long time but has a very low risk of discovery before success. We need not impute impatience or hyperbolic discounting to the AI.
But here I’ll grant we are well adrift in to groundless and fruitless speculation: we don’t and can’t have anything like the information needed to guess at what strategy would look best.
Anyway, the assumption that an AI could understand human motivation, and become a skilled manipulator, is already too far-fetched for me to take seriously.
I wouldn’t say I’m taking the idea seriously either—more taking it for a ride. I share much of your skepticism here. I don’t think we can say that it’s impossible to make an AI with advanced social intelligence, but I think we can say that it is very unlikely to be achievable in the near to medium term.
This is a separate question from the one asked in the OP, though.
I was treating it as part of the premises of the discussion that the AI is at least indifferent to doing so: it needs only enough infrastructure left for it to continue to exist and be able to rebuild under its own total control.
How many humans does it take to keep the infrastructure running that is necessary to create new and better CPU’s etc.? I am highly confident that it takes more than the random patches of civilization left over after deploying a bioweapon on a global scale.
Surely we can imagine a science fiction world in which the AI has access to nanoassemblers, or in which the world’s infrastructure is maintained by robot drones. But then, what do we have? We have a completely artificial scenario designed to yield the desired conclusion. An AI with some set of vague abilities, and circumstances under which these abilities suffice to take over the world.
As I wrote several times in the past. If your AI requires nanotechnology, bioweapons, or a fragile world, then superhuman AI is our least worry, because long before we will create it, the tools necessary to create it will allow unfriendly humans to do the same.
Bioweapons: If an AI can use bioweapons to blackmail the world into submission, then some group of people will be able to do that before this AI is created (dispatch members in random places around the world).
Nanotechnology: It seems likely to me that narrow AI precursors will suffice in order for humans to create nanotechnology. Which makes it a distinct risk.
A fragile world: I suspect that a bunch of devastating cyber-attacks and wars will be fought before the first general AI capable of doing the same. Governments will realize that their most important counterstrike resources need to be offline. In other words, it seems very unlikely that an open confrontation with humans would be a viable strategy for a fragile high-tech product such as the first general AI. And taking over a bunch of refrigerators, mobile phones and cars is only a catastrophic risk, not an existential one.
I really don’t think we have to posit nanoassemblers for this particular scenario to work. Robot drones are needed, but I think they fall out as a consequence of currently existing robots and the all-singing all-dancing AI we’ve imagined in the first place. There are shedloads of robots around at the moment—the OP mentioned the existence of Internet-connected robot-controlled cars, but there are plenty of others, including most high tech manufacturer. Sure, those robots aren’t autonomous, but they don’t need to be if we’ve assumed an all-singing all-dancing AI in the first place. I think that might be enough to keep the power and comms on in a few select areas with a bit of careful planning.
Rebuilding/restarting enough infrastructure to be able to make new and better CPUs (and new and better robot extensions of the AI) would take an awfully long time, granted, but the AI is free of human threat at that point.
Do you believe that if Obama were to ask the NSA to take over Russia, that the NSA could easily do so? If so, I am speechless.
Ordering the NSA to take over Russia would effectively result in WWIII.
Anyway, the assumption that an AI could understand human motivation, and become a skilled manipulator, is already too far-fetched for me to take seriously.
For what values of skill do you believe that to be true? Do you think there are reason to believe that an AGI who is online won’t be as good at manipulating as the best humans?
For the AI-box scenario I can understand if you think that the AGI doesn’t have enough interactions with humans to train a decent model of human motivation to be good at manipulating.
Do you believe that if Obama were to ask the NSA to take over Russia, that the NSA could easily do so? If so, I am speechless.
Let’s look at one of the most realistic schemes, creating a bioweapon. Yes, an organization like the NSA could probably design such a bioweapon. But how exactly could they take over the world that way?
They could either use the bioweapon to kill a huge number of people, or use it to blackmail the world into submission. I believe that the former would cause our technological civilization, on which the NSA depends, to collapse. So that would be stupid. The latter would maybe work for some time, until the rest of the world got together, in order to make a believable threat of mutual destruction.
I just don’t see this to be a viable way to take over the world. At least not in such a way that you would gain actual control.
Now I can of course imagine a different world, in which it would be possible to gain control. Such as a world in which everyone important was using advanced brain implants. If these brain implants could be hacked, even the NSA could take over the world. That’s a no-brainer.
I can also imagine a long-term plan. But those are very risky. The longer it takes, the higher the chance that your plan is revealed. Also, other AI’s, with different, opposing utility-functions, will be employed. Some will be used to detect such plans.
Anyway, the assumption that an AI could understand human motivation, and become a skilled manipulator, is already too far-fetched for me to take seriously. People around here too often confound theory with practice. That all this might be physically possible does not prove that it is at all likely.
No. I think the phrase “take over” is describing two very different scenarios if we compare “Obama trying to take over the world” and “a hypothetical hostile AI trying to take over the world”. Obama has many human scruples and cares a lot about continued human survival, and specifically not just about the continued existence of the people of the USA but that they thrive. (Thankfully!)
I entirely agree that killing huge numbers of people would be a stupid thing for the actual NSA and/or Obama to do. Killing all the people, themselves included, would not only fail to achieve any of their goals but thwart (almost) all of them permanently. I was treating it as part of the premises of the discussion that the AI is at least indifferent to doing so: it needs only enough infrastructure left for it to continue to exist and be able to rebuild under its own total control.
Yes, indeed, the longer it takes the higher the chance that the plan is revealed. But a different plan may take longer but still have a lower overall chance of failure if its risk of discovery per unit time is substantially lower. Depending on the circumstances, one can imagine an AI calculating that its best interests lie in a plan that takes a very long time but has a very low risk of discovery before success. We need not impute impatience or hyperbolic discounting to the AI.
But here I’ll grant we are well adrift in to groundless and fruitless speculation: we don’t and can’t have anything like the information needed to guess at what strategy would look best.
I wouldn’t say I’m taking the idea seriously either—more taking it for a ride. I share much of your skepticism here. I don’t think we can say that it’s impossible to make an AI with advanced social intelligence, but I think we can say that it is very unlikely to be achievable in the near to medium term.
This is a separate question from the one asked in the OP, though.
How many humans does it take to keep the infrastructure running that is necessary to create new and better CPU’s etc.? I am highly confident that it takes more than the random patches of civilization left over after deploying a bioweapon on a global scale.
Surely we can imagine a science fiction world in which the AI has access to nanoassemblers, or in which the world’s infrastructure is maintained by robot drones. But then, what do we have? We have a completely artificial scenario designed to yield the desired conclusion. An AI with some set of vague abilities, and circumstances under which these abilities suffice to take over the world.
As I wrote several times in the past. If your AI requires nanotechnology, bioweapons, or a fragile world, then superhuman AI is our least worry, because long before we will create it, the tools necessary to create it will allow unfriendly humans to do the same.
Bioweapons: If an AI can use bioweapons to blackmail the world into submission, then some group of people will be able to do that before this AI is created (dispatch members in random places around the world).
Nanotechnology: It seems likely to me that narrow AI precursors will suffice in order for humans to create nanotechnology. Which makes it a distinct risk.
A fragile world: I suspect that a bunch of devastating cyber-attacks and wars will be fought before the first general AI capable of doing the same. Governments will realize that their most important counterstrike resources need to be offline. In other words, it seems very unlikely that an open confrontation with humans would be a viable strategy for a fragile high-tech product such as the first general AI. And taking over a bunch of refrigerators, mobile phones and cars is only a catastrophic risk, not an existential one.
I really don’t think we have to posit nanoassemblers for this particular scenario to work. Robot drones are needed, but I think they fall out as a consequence of currently existing robots and the all-singing all-dancing AI we’ve imagined in the first place. There are shedloads of robots around at the moment—the OP mentioned the existence of Internet-connected robot-controlled cars, but there are plenty of others, including most high tech manufacturer. Sure, those robots aren’t autonomous, but they don’t need to be if we’ve assumed an all-singing all-dancing AI in the first place. I think that might be enough to keep the power and comms on in a few select areas with a bit of careful planning.
Rebuilding/restarting enough infrastructure to be able to make new and better CPUs (and new and better robot extensions of the AI) would take an awfully long time, granted, but the AI is free of human threat at that point.
Ordering the NSA to take over Russia would effectively result in WWIII.
For what values of skill do you believe that to be true? Do you think there are reason to believe that an AGI who is online won’t be as good at manipulating as the best humans?
For the AI-box scenario I can understand if you think that the AGI doesn’t have enough interactions with humans to train a decent model of human motivation to be good at manipulating.