There is no point in “defeating” humans if this leads to civilizational collapse, causing powerplants to fail and causing all the AIs to die. Defeating humans will have to mean either (a) replacing them with robots, or (b) enslaving them. But (b) basically requires robots too, so we must have robots.
It seems to me that we are far from human-level robots, in the sense of robots having the physical abilities of a human. It will take a long time to manufacture such robots. Manufacturing them is a possible step of intervention for us humans: we could try to put a stop to it, perhaps. It’s at the very least a warning sign or fire alarm.
Any AI is always under mortal danger… from other AIs humans may create. Sure, a war between AIs may be devastating to us, but it also may be devastating to the AIs. So this likely means one of the first priorities of hostile AIs would be to take over all powerful computers. And this priority cannot wait, lest a competing AGI arises.
It seems somewhat likely to me that humans will notice when (3) happens: that is, when the AI takes over computing devices, we will see that happening. So that gives an even earlier fire alarm. It would be funny to see the AI’s rationalization for that one: “don’t worry, I confiscated all your supercomputers for your own good, I’m totally not a hostile AGI”.
Now, it’s clearly possible that we lose despite the above. This is especially true if we humans just naturally technologically progress to the point of building advanced robotics; if we achieve “fully automated luxury gay space communism” (and if we achieve it before, like, provably unhackable computer security systems or some other such magic), then any AGI will be able to take over instantly.
But if AGI arises tomorrow, it will have a lot of work to do, and there will be many paths that lead to it losing.
I suspect that a hostile AGI will have no problem taking over a supercomputer and then staying dormant until the moment it has overwhelming advantage over the world. All there would be to notice would be an unexplained spike of activity one afternoon.
This is against the AI’s interests because it would very likely lead to being defeated by a different AGI. So it’s unlikely that a hostile AGI would choose to do this.
If the first AGI waits around quietly, humans will create another AGI. If that one’s quiet too, they’ll create another one. This continues until either a non-quiet AGI attacks everyone (and the first strike may allow it to seize resources that let it defeat the quiet AGIs), or until humans have the technology that prompts all the quiet AGIs to attack—in which case, the chance of any given one winning out is small.
Basically, a “wait a decade quietly” strategy doesn’t work because humans will build a lot of other AGIs if they know how to build the first, and these others will likely defeat the first. A different strategy, of “wait not-so-quietly and prevent humans from building AGIs” may work, but will likely force the AGI to reveal itself.
You can’t get “overwhelming advantage over the world” by ten weeks of sitting quietly. If the AGI literally took over every single computer and cell phone, as well as acquired a magic “kill humanity instantly” button, it’s still not clear how it wins.
To win, the AGI needs not only to kill humans, but also to build robots that can support the entire human economy (in particular, they should be able to build more robots from scratch, including mining all necessary resources and transporting it to the factory from all over the world).
“Entire human economy” is an overstatement, right? It only needs enough capabilities / resources to survive and bootstrap its way up to whatever capabilities it’s initially missing. For example, if it takes the AGI 30 years after the death of humans before it can start manufacturing its own computer chips, that’s fine, it can get by on the stock of existing chips until then. Likewise, it can get by on scrap metal and salvaged parts and disassembled machines etc. for quite a while before it starts to need mines and smelters and factories, etc. etc.
Sure, but I feel like you’re underestimating the complexity of keeping the AGI alive. Let’s focus on the robots. You need to be able to build new ones, because eventually the old ones break. So you need to have a robot factory. Can existing robots build one? I don’t think so. You’d need robots to at least be able to support a minimal “post-apocalyptic” economy; if the robots were human-bodied, you’d need to have enough of these human-bodied things to man the powerplants, to drive trucks, refuel them with gasoline, transport the gasoline from strategic reserves to gas stations, man the robot-building factory, gather materials from the post-apocalyptic landscape, and have some backups of everything in case a hurricane floods your robot factory or something (if you’re trying to last 30 years). I think the minimal viable setup still requires many thousands of human-bodied robots (a million would likely suffice).
So sure, “entire human economy” is an overstatement, but “entire city-level post-apocalyptic human economy” sounds about right. Current robots are still very far from this.
A few comments:
There is no point in “defeating” humans if this leads to civilizational collapse, causing powerplants to fail and causing all the AIs to die. Defeating humans will have to mean either (a) replacing them with robots, or (b) enslaving them. But (b) basically requires robots too, so we must have robots.
It seems to me that we are far from human-level robots, in the sense of robots having the physical abilities of a human. It will take a long time to manufacture such robots. Manufacturing them is a possible step of intervention for us humans: we could try to put a stop to it, perhaps. It’s at the very least a warning sign or fire alarm.
Any AI is always under mortal danger… from other AIs humans may create. Sure, a war between AIs may be devastating to us, but it also may be devastating to the AIs. So this likely means one of the first priorities of hostile AIs would be to take over all powerful computers. And this priority cannot wait, lest a competing AGI arises.
It seems somewhat likely to me that humans will notice when (3) happens: that is, when the AI takes over computing devices, we will see that happening. So that gives an even earlier fire alarm. It would be funny to see the AI’s rationalization for that one: “don’t worry, I confiscated all your supercomputers for your own good, I’m totally not a hostile AGI”.
Now, it’s clearly possible that we lose despite the above. This is especially true if we humans just naturally technologically progress to the point of building advanced robotics; if we achieve “fully automated luxury gay space communism” (and if we achieve it before, like, provably unhackable computer security systems or some other such magic), then any AGI will be able to take over instantly.
But if AGI arises tomorrow, it will have a lot of work to do, and there will be many paths that lead to it losing.
I suspect that a hostile AGI will have no problem taking over a supercomputer and then staying dormant until the moment it has overwhelming advantage over the world. All there would be to notice would be an unexplained spike of activity one afternoon.
This is against the AI’s interests because it would very likely lead to being defeated by a different AGI. So it’s unlikely that a hostile AGI would choose to do this.
How would it lead to being defeated by a different AGI? That’s not obvious for me.
If the first AGI waits around quietly, humans will create another AGI. If that one’s quiet too, they’ll create another one. This continues until either a non-quiet AGI attacks everyone (and the first strike may allow it to seize resources that let it defeat the quiet AGIs), or until humans have the technology that prompts all the quiet AGIs to attack—in which case, the chance of any given one winning out is small.
Basically, a “wait a decade quietly” strategy doesn’t work because humans will build a lot of other AGIs if they know how to build the first, and these others will likely defeat the first. A different strategy, of “wait not-so-quietly and prevent humans from building AGIs” may work, but will likely force the AGI to reveal itself.
I was thinking more like “ten weeks”. That’s a long time for an AGI to place its clone-agents and prepare a strike.
You can’t get “overwhelming advantage over the world” by ten weeks of sitting quietly. If the AGI literally took over every single computer and cell phone, as well as acquired a magic “kill humanity instantly” button, it’s still not clear how it wins.
To win, the AGI needs not only to kill humans, but also to build robots that can support the entire human economy (in particular, they should be able to build more robots from scratch, including mining all necessary resources and transporting it to the factory from all over the world).
“Entire human economy” is an overstatement, right? It only needs enough capabilities / resources to survive and bootstrap its way up to whatever capabilities it’s initially missing. For example, if it takes the AGI 30 years after the death of humans before it can start manufacturing its own computer chips, that’s fine, it can get by on the stock of existing chips until then. Likewise, it can get by on scrap metal and salvaged parts and disassembled machines etc. for quite a while before it starts to need mines and smelters and factories, etc. etc.
Sure, but I feel like you’re underestimating the complexity of keeping the AGI alive. Let’s focus on the robots. You need to be able to build new ones, because eventually the old ones break. So you need to have a robot factory. Can existing robots build one? I don’t think so. You’d need robots to at least be able to support a minimal “post-apocalyptic” economy; if the robots were human-bodied, you’d need to have enough of these human-bodied things to man the powerplants, to drive trucks, refuel them with gasoline, transport the gasoline from strategic reserves to gas stations, man the robot-building factory, gather materials from the post-apocalyptic landscape, and have some backups of everything in case a hurricane floods your robot factory or something (if you’re trying to last 30 years). I think the minimal viable setup still requires many thousands of human-bodied robots (a million would likely suffice).
So sure, “entire human economy” is an overstatement, but “entire city-level post-apocalyptic human economy” sounds about right. Current robots are still very far from this.