You can’t get “overwhelming advantage over the world” by ten weeks of sitting quietly. If the AGI literally took over every single computer and cell phone, as well as acquired a magic “kill humanity instantly” button, it’s still not clear how it wins.
To win, the AGI needs not only to kill humans, but also to build robots that can support the entire human economy (in particular, they should be able to build more robots from scratch, including mining all necessary resources and transporting it to the factory from all over the world).
“Entire human economy” is an overstatement, right? It only needs enough capabilities / resources to survive and bootstrap its way up to whatever capabilities it’s initially missing. For example, if it takes the AGI 30 years after the death of humans before it can start manufacturing its own computer chips, that’s fine, it can get by on the stock of existing chips until then. Likewise, it can get by on scrap metal and salvaged parts and disassembled machines etc. for quite a while before it starts to need mines and smelters and factories, etc. etc.
Sure, but I feel like you’re underestimating the complexity of keeping the AGI alive. Let’s focus on the robots. You need to be able to build new ones, because eventually the old ones break. So you need to have a robot factory. Can existing robots build one? I don’t think so. You’d need robots to at least be able to support a minimal “post-apocalyptic” economy; if the robots were human-bodied, you’d need to have enough of these human-bodied things to man the powerplants, to drive trucks, refuel them with gasoline, transport the gasoline from strategic reserves to gas stations, man the robot-building factory, gather materials from the post-apocalyptic landscape, and have some backups of everything in case a hurricane floods your robot factory or something (if you’re trying to last 30 years). I think the minimal viable setup still requires many thousands of human-bodied robots (a million would likely suffice).
So sure, “entire human economy” is an overstatement, but “entire city-level post-apocalyptic human economy” sounds about right. Current robots are still very far from this.
I was thinking more like “ten weeks”. That’s a long time for an AGI to place its clone-agents and prepare a strike.
You can’t get “overwhelming advantage over the world” by ten weeks of sitting quietly. If the AGI literally took over every single computer and cell phone, as well as acquired a magic “kill humanity instantly” button, it’s still not clear how it wins.
To win, the AGI needs not only to kill humans, but also to build robots that can support the entire human economy (in particular, they should be able to build more robots from scratch, including mining all necessary resources and transporting it to the factory from all over the world).
“Entire human economy” is an overstatement, right? It only needs enough capabilities / resources to survive and bootstrap its way up to whatever capabilities it’s initially missing. For example, if it takes the AGI 30 years after the death of humans before it can start manufacturing its own computer chips, that’s fine, it can get by on the stock of existing chips until then. Likewise, it can get by on scrap metal and salvaged parts and disassembled machines etc. for quite a while before it starts to need mines and smelters and factories, etc. etc.
Sure, but I feel like you’re underestimating the complexity of keeping the AGI alive. Let’s focus on the robots. You need to be able to build new ones, because eventually the old ones break. So you need to have a robot factory. Can existing robots build one? I don’t think so. You’d need robots to at least be able to support a minimal “post-apocalyptic” economy; if the robots were human-bodied, you’d need to have enough of these human-bodied things to man the powerplants, to drive trucks, refuel them with gasoline, transport the gasoline from strategic reserves to gas stations, man the robot-building factory, gather materials from the post-apocalyptic landscape, and have some backups of everything in case a hurricane floods your robot factory or something (if you’re trying to last 30 years). I think the minimal viable setup still requires many thousands of human-bodied robots (a million would likely suffice).
So sure, “entire human economy” is an overstatement, but “entire city-level post-apocalyptic human economy” sounds about right. Current robots are still very far from this.