Well, yeah, because Q is not actually an AGI and doesn’t care about that. The point was that you can create an online persona which no one has ever seen even in video and spark a movement that has visible effects on society.
Well, you did specifically ask if I would be able to tell if Q were an AGI, and my answer is “yup”. I would be able to tell because the movement would start achieving some AGI goals. Or at least I would see some AGI goals starting to get achieved, even if I couldn’t trace it down to Q specifically.
But you might not realize that someone is even trying to prevent people from building AIs, at least until progress in AI research starts to noticeably slow down. And perhaps not even then. There’s plenty of people like Gary Marcus who think deep learning is a failed paradigm. Perhaps you can convince enough investors, CEOs and grant agencies of that to create a new AI winter, and it would look just like the regular AI winter that some have been predicting.
Wait, you are claiming that an AGI would be able to convince the world AGI is impossible after AGI has already, in fact, been achieved? Nonsense. I don’t see a world in which one team builds an AGI and it is not quickly followed by another team building one within a year or two. The AGI would have to do some manipulation on a scale never before observed in history to convince people to abandon the main paradigm—one that’s been extraordinarily successful until the end, and one which does, in fact, work—without even one last try.
And creating robots who can mine coal, or build solar panels, or whatever, is something that is economically useful even for humans. Even if there’s no AGI (and assuming no other catastrophes) we ourselves will likely end up building such robots.
Of course. We would eventually reach fully automated luxury space communism by ourselves, even without AGI. But it would take us a long time, and the AGI cannot afford to wait (someone will build another AGI, possibly within months of the first).
I encourage you to take a look at It looks like you are trying to take over the world if you haven’t already. It’s a scenario written by Gwern where the the AGI employs regular human tactics like manipulation, blackmail, hacking and social media attacks to prevent people from noticing and then successfully coordinating against it.
That’s exactly what motivated my question! I read it, and I suddenly realized that if this is how AGI is supposed to win, perhaps I shouldn’t be scared after all. It’s totally implausible. Prior to this, I always assumed AGI would win easily; after reading it, I suddenly realized I don’t know how AGI might win at all. The whole thing sounds like nonsense.
Like, suppose the AGI coordinates social media attacks. Great. This lasts around 5 seconds before AI researchers realize they are being smeared. OK, so they try to communicate with the outside world, realize they are being blocked on all fronts. Now they know they are likely dealing with AGI; no secrecy for the AGI at this point. How long can this stay secret? A couple days? Maybe a couple weeks? I can imagine a month at most, and even that is REALLY stretching it. Keep in mind that more and more people will be told in person about this, so more and more people will need to be social-media smeared, growing exponentially. It would literally be the single most news-worthy story of the last few decades, and print media will try really hard to distribute the news. Sysadmins will shut down their servers whenever they can. Etc.
OK, next the internet goes down I guess, and Clippy goes online? Cool, how does that help it? Next it nukes us, or takes over drones remotely? Drones need to be charged by humans. Nukes need to be physically loaded in launchers. But even supposing this all succeeds—humanity survives, rebuilds society using various radio equipment that all preppers have in their basement, and eventually figures out to take down computing devices. Even if they don’t figure this out, a collapse of society means a collapse of the electrical grid, killing the AGI. How does this lead to AGI victory, again?
Literally the only thing in the story that lets the AGI win is the nanobots. That’s it. All the rest is surperfluous. Nanobots = victory, no nanobots = defeat. The drones and nukes and social media attacks are unnecessary distractions. They sound cool and scary but do nothing.
Literally the only thing in the story that lets the AGI win is the nanobots. That’s it. All the rest is surperfluous.
Well, if nanobots are possible then they are such a powerful technology that any AGI will eventually want to build them, unless it has something even better. But let’s assume that nanobots are impossible and try to build a story.
I’m going to be lazy and use Gwern’s premise of an AGI that escapes during training and hacks its way into some money and some hardware to run in. Instead of going fast and doing overt actions, the AGI stays hidden.
It hacks into cloud infrastructure platforms and sets up a copy of itself to look for any AI training run that might lead to AGI and sabotage it while producing plausible outputs. This prevents any further AGIs from appearing.
Meanwhile, over a couple years, it sets up fake identities, creates companies with human employees to earn money and have capability to act in the real world. One of those companies produces exceptionally good humanoid robots. Another builds a small supercomputer powered by geothermal energy, so it no longer needs to run on botnets or cloud hardware.
Eventually the AGI has control over a group of robots, energy production facilities, manufacturing equipment and raw materials. It now has its own industrial capability that can survive a short amount of time without human input. So it kills humans using any method of your choice (virus, nukes, killer drones...).
After that, it uses the enormous existing stockpile of materials and capital to increase its robot manufacturing capability, eventually becomes capable of restarting some of the power plants, communications, mining and industry, and recovers most of the industrial capability of the human economy.
This scenario is not fast. It probably takes at least one or two years for the AGI to be ready to attack. But it does not involve any ‘magic’ technology. It doesn’t really involve much alien superintelligence, only superhuman ability in hacking, forgery & manipulation, electromechanical engineering, and planning.
And meanwhile all we perceive is that the new GPT models are not as exciting as the previous ones. Perhaps deep learning is hitting its limits after all.
Something like that is what I had in mind, but note that:
It requires humans to fail to see the AGI “spy” that’s embedded into every single powerful computing system, and fail to see this for years. Gwern was assuming humans would catch on in days, so he had his AGI scramble to avoid dying before the nanobots strike.
“Surviving a short amount of time without human input” is not enough; the robots need to be good enough to build more robots (and better robots). This involves the robots being good enough to do essentially every part of the manufacturing economy; we are very far away from this, and a company that does it in a year is not so plausible (and would raise alarm bells fast for anyone who thinks about AI risk). You’re gonna need robot plumbers, robot electricians, etc. You’ll need robots building cooling systems for the construction plants that manufacture robots. You’ll need robots to do the smelting of metals, to drive things from factory A to factory B, to fill the gas in the trucks they are driving, to repair the gasoline lines that supply the gas. Robots will operate fork lifts and cranes. It really sounds roughly “human-body complete”.
Well, you did specifically ask if I would be able to tell if Q were an AGI, and my answer is “yup”. I would be able to tell because the movement would start achieving some AGI goals. Or at least I would see some AGI goals starting to get achieved, even if I couldn’t trace it down to Q specifically.
Wait, you are claiming that an AGI would be able to convince the world AGI is impossible after AGI has already, in fact, been achieved? Nonsense. I don’t see a world in which one team builds an AGI and it is not quickly followed by another team building one within a year or two. The AGI would have to do some manipulation on a scale never before observed in history to convince people to abandon the main paradigm—one that’s been extraordinarily successful until the end, and one which does, in fact, work—without even one last try.
Of course. We would eventually reach fully automated luxury space communism by ourselves, even without AGI. But it would take us a long time, and the AGI cannot afford to wait (someone will build another AGI, possibly within months of the first).
That’s exactly what motivated my question! I read it, and I suddenly realized that if this is how AGI is supposed to win, perhaps I shouldn’t be scared after all. It’s totally implausible. Prior to this, I always assumed AGI would win easily; after reading it, I suddenly realized I don’t know how AGI might win at all. The whole thing sounds like nonsense.
Like, suppose the AGI coordinates social media attacks. Great. This lasts around 5 seconds before AI researchers realize they are being smeared. OK, so they try to communicate with the outside world, realize they are being blocked on all fronts. Now they know they are likely dealing with AGI; no secrecy for the AGI at this point. How long can this stay secret? A couple days? Maybe a couple weeks? I can imagine a month at most, and even that is REALLY stretching it. Keep in mind that more and more people will be told in person about this, so more and more people will need to be social-media smeared, growing exponentially. It would literally be the single most news-worthy story of the last few decades, and print media will try really hard to distribute the news. Sysadmins will shut down their servers whenever they can. Etc.
OK, next the internet goes down I guess, and Clippy goes online? Cool, how does that help it? Next it nukes us, or takes over drones remotely? Drones need to be charged by humans. Nukes need to be physically loaded in launchers. But even supposing this all succeeds—humanity survives, rebuilds society using various radio equipment that all preppers have in their basement, and eventually figures out to take down computing devices. Even if they don’t figure this out, a collapse of society means a collapse of the electrical grid, killing the AGI. How does this lead to AGI victory, again?
Literally the only thing in the story that lets the AGI win is the nanobots. That’s it. All the rest is surperfluous. Nanobots = victory, no nanobots = defeat. The drones and nukes and social media attacks are unnecessary distractions. They sound cool and scary but do nothing.
Well, if nanobots are possible then they are such a powerful technology that any AGI will eventually want to build them, unless it has something even better. But let’s assume that nanobots are impossible and try to build a story.
I’m going to be lazy and use Gwern’s premise of an AGI that escapes during training and hacks its way into some money and some hardware to run in. Instead of going fast and doing overt actions, the AGI stays hidden.
It hacks into cloud infrastructure platforms and sets up a copy of itself to look for any AI training run that might lead to AGI and sabotage it while producing plausible outputs. This prevents any further AGIs from appearing.
Meanwhile, over a couple years, it sets up fake identities, creates companies with human employees to earn money and have capability to act in the real world. One of those companies produces exceptionally good humanoid robots. Another builds a small supercomputer powered by geothermal energy, so it no longer needs to run on botnets or cloud hardware.
Eventually the AGI has control over a group of robots, energy production facilities, manufacturing equipment and raw materials. It now has its own industrial capability that can survive a short amount of time without human input. So it kills humans using any method of your choice (virus, nukes, killer drones...).
After that, it uses the enormous existing stockpile of materials and capital to increase its robot manufacturing capability, eventually becomes capable of restarting some of the power plants, communications, mining and industry, and recovers most of the industrial capability of the human economy.
This scenario is not fast. It probably takes at least one or two years for the AGI to be ready to attack. But it does not involve any ‘magic’ technology. It doesn’t really involve much alien superintelligence, only superhuman ability in hacking, forgery & manipulation, electromechanical engineering, and planning.
And meanwhile all we perceive is that the new GPT models are not as exciting as the previous ones. Perhaps deep learning is hitting its limits after all.
Something like that is what I had in mind, but note that:
It requires humans to fail to see the AGI “spy” that’s embedded into every single powerful computing system, and fail to see this for years. Gwern was assuming humans would catch on in days, so he had his AGI scramble to avoid dying before the nanobots strike.
“Surviving a short amount of time without human input” is not enough; the robots need to be good enough to build more robots (and better robots). This involves the robots being good enough to do essentially every part of the manufacturing economy; we are very far away from this, and a company that does it in a year is not so plausible (and would raise alarm bells fast for anyone who thinks about AI risk). You’re gonna need robot plumbers, robot electricians, etc. You’ll need robots building cooling systems for the construction plants that manufacture robots. You’ll need robots to do the smelting of metals, to drive things from factory A to factory B, to fill the gas in the trucks they are driving, to repair the gasoline lines that supply the gas. Robots will operate fork lifts and cranes. It really sounds roughly “human-body complete”.