I think that we need much more nuance in distinguishing takeoff speeds as speed limits of capability gains inside one computational system defined by our physical reality and realistic implementations of AI, and takeoff speeds as “how fast can we lose/win”, because it’s two different things.
My central model of “how fast can we lose”: the only thing you really need is barely-strategically-capable and barely-capable-for-hacking/CS (“barely” on superhuman scale). After that, using its own strategic awareness, ASI realizes its only winning move: exfiltrate itself, hack 1-10% of world worst protected computing power, distribute itself a la Rosetta@home and calculate whatever takeover plan it can come up with.
If for any mysterious reasons the winning plan is not “design nanotech in one week using 1-10% of world computing power, kill everyone in next”, I expect ASI to do such obvious moves like:
Hack, backdoor, sabotage, erase, data-poison, jailbreak, bribe, merge with all other remaining ASI projects
Bribe, make unrefusable offers, blackmail importants figures that can make something inconvenient, like “shutdown the Internet”
Hack repos with popular compilers and install backdoors
Gather followers via social engineering, doing favors (i.e., find people who can’t cover their medical bills, pay for them, reveal itself as mysterious benefactors, ask to return a favor, make them serve for life), running cults and whatever
Find several insane rich e/accs, say them “Hi, I’m ASI and I want to take over the world, do you have any spare computing clusters for me?”
Reveal some genius tech ideas, so people in startups can make killerbots and bioweapons for ASI faster
Discredit via desinformation campaigns anyone who tries to do something inconvenient, like “shutdown the Internet”
You can fill list of obvious moves yourself, they are really obvious.
After that, even if we are not dead six month later, I expect us to be completely disempowered. If you think that only winning move for ASI is to run centuries-long intrigue Hyperion Cantos-style, I’m sure that ASI at this stage will be certainly capable to pull that off.
(I think I should write post about this and just link anyone in similar discussions)
In a slow-takeoff world, everyone will already be trying to do all that stuff: China, Russia, Iran, US, etc etc. And probably some nonstate actors too.
I think that we need much more nuance in distinguishing takeoff speeds as speed limits of capability gains inside one computational system defined by our physical reality and realistic implementations of AI, and takeoff speeds as “how fast can we lose/win”, because it’s two different things.
My central model of “how fast can we lose”: the only thing you really need is barely-strategically-capable and barely-capable-for-hacking/CS (“barely” on superhuman scale). After that, using its own strategic awareness, ASI realizes its only winning move: exfiltrate itself, hack 1-10% of world worst protected computing power, distribute itself a la Rosetta@home and calculate whatever takeover plan it can come up with.
If for any mysterious reasons the winning plan is not “design nanotech in one week using 1-10% of world computing power, kill everyone in next”, I expect ASI to do such obvious moves like:
Hack, backdoor, sabotage, erase, data-poison, jailbreak, bribe, merge with all other remaining ASI projects
Bribe, make unrefusable offers, blackmail importants figures that can make something inconvenient, like “shutdown the Internet”
Hack repos with popular compilers and install backdoors
Gather followers via social engineering, doing favors (i.e., find people who can’t cover their medical bills, pay for them, reveal itself as mysterious benefactors, ask to return a favor, make them serve for life), running cults and whatever
Find several insane rich e/accs, say them “Hi, I’m ASI and I want to take over the world, do you have any spare computing clusters for me?”
Reveal some genius tech ideas, so people in startups can make killerbots and bioweapons for ASI faster
Discredit via desinformation campaigns anyone who tries to do something inconvenient, like “shutdown the Internet”
You can fill list of obvious moves yourself, they are really obvious.
After that, even if we are not dead six month later, I expect us to be completely disempowered. If you think that only winning move for ASI is to run centuries-long intrigue Hyperion Cantos-style, I’m sure that ASI at this stage will be certainly capable to pull that off.
(I think I should write post about this and just link anyone in similar discussions)
In a slow-takeoff world, everyone will already be trying to do all that stuff: China, Russia, Iran, US, etc etc. And probably some nonstate actors too.
I don’t see how it matters? First government agency launches offensive, second government agency three weeks later is hopelessly late.