Is there any way for the AI to take over the world OTHER THAN nanobots? Every time taking over the world comes up, people just say “nanobots”. OK. Is there anything else?
Note that killing all humans is not sufficient; this is a fail condition for the AGI. If you kill all humans, nobody mines natural gas anymore, so no power grid, and the AGI dies. The AGI needs to replace humans with advanced robots, and do so before all power goes down. Nanobots can do this if they are sufficiently advanced, but “virus that kills all humans” is insufficient and leads to the AGI’s death.
So, again, anything other than nanobots? Because I’m not sure nanobots are plausible. I don’t think you can build them just by paying someone to mix proteins—I doubt you could even form a single functional cell that way, even of a known organism like a bacteria. Then there is the issue that the biological world is very complicated and predicting the behavior of the nanobots in real-world environments is likely difficult. Then there is also the issue that simulating proteins (or other chemicals) at very high fidelities is fundamentally a quantum mechanical problem, and would require quantum computers.
I would say “advanced memetics”. Like “AGI uploads weird video on Youtube, it goes viral, 3 billions people watch it and do what AGI needs them to do from now on, for example, build robots and commit suicide when there are enough robots. All AI and AI Safety researchers are subjected to a personalized memetic attack, of course”.
This is a really, really implausible scenario again. You have no evidence that such memetics exist, and the smart money is that they don’t. If they do, there’s no guarantee that the AI would be able to figure them out. Being smarter than humans—even way smarter than humans—does not equate to godhood. The AI will not be able to predict the weather 3 weeks out, and I’m not sure that it will be able to predict the exact reactions of each of a billion different human brain to a video input—not at the granularity required for something like what you’re suggesting.
I think AI is a threat. I’m trying to be on your side here. But I really can’t swallow these exaggerated, made up scenarios.
It’s somewhat easier to think of scenarios where the takeover happens slowly.
There’s the whole “ascended economy” scenarios where AGI deceptively convinces everyone that it is aligned or narrow, is deployed gradually in more and more domains, automates more and more parts of the economy using regular robots until humans are not needed anymore, and then does the lethal virus thing or defects in other way.
There’s the scenario where the AGI uploads itself into the cloud, uses hacking/manipulation/financial prowess to sustain itself, then uses manipulation to slowly poison our collective epistemic process, gaining more and more power. How much influence does QAnon have? If Q was an AGI posting on 4chan instead of a human, would you be able to tell? What about Satoshi Nakamoto?
Non-nanobot scenarios where the AGI quickly gains power are a bit harder to imagine, but a fertile source of those might be something like the AGI convincing a lot of people that it’s some kind of prophet. Then uses its follower base to gain power over the real world.
If merely human dictators manage to get control over whole countries all the time, I think it’s quite plausible that a superintelligence could to do the same with the whole world. Even without anyone noticing that they’re dealing with a superintelligence.
And look at Yudkowsky himself, who played a very significant role in getting very talented people to dedicate their lives and their billions to EA / AI safety, mostly by writing in a way that is extremely appealing to a certain set of people. I sometimes joke that HPMOR overwrote my previous personality. I’m sure a sufficiently competent AGI can do much more.
If Q was an AGI posting on 4chan instead of a human, would you be able to tell?
That would be incredibly risky for the AGI, since Q has done nothing to prevent another AGI from being built. The most important concern an AGI must deal with is that humans can build another AGI, and pulling a Satoshi or a QAnon does nothing to address this.
If merely human dictators manage to get control over whole countries all the time, I think it’s quite plausible that a superintelligence could to do the same with the whole world. Even without anyone noticing that they’re dealing with a superintelligence.
I personally would likely notice: anyone who successfully prevents people from building AIs is a high suspect of being an AGI themselves. Anyone who causes the creation of robots who can mine coal or something (to generate electricity without humans) is likely an AGI themselves. That doesn’t mean I’d be able to stop them, necessarily. I’m just saying, “nobody would notice” is a stretch.
I sometimes joke that HPMOR overwrote my previous personality.
I agree that the AGI could build a cultish following like Yudkowsky did.
Q has done nothing to prevent another AGI from being built
Well, yeah, because Q is not actually an AGI and doesn’t care about that. The point was that you can create an online persona which no one has ever seen even in video and spark a movement that has visible effects on society.
The most important concern an AGI must deal with is that humans can build another AGI, and pulling a Satoshi or a QAnon does nothing to address this.
Even if two or more AGIs end up competing among themselves, this does not imply that we survive. It probably looks more like European states dividing Africa among themselves while constantly fighting each other.
And pulling a Satoshi or a QAnon can definitely do something to address that. You can buy a lot of hardware to drive up prices and discourage building more datacenters for training AI. You can convince people to carry out terrorist attacks againts chip fabs. You can offer top AI researchers huge amounts of money to work on some interesting problem that you know to be a dead-end approach.
I personally would likely notice: anyone who successfully prevents people from building AIs is a high suspect of being an AGI themselves. Anyone who causes the creation of robots who can mine coal or something (to generate electricity without humans) is likely an AGI themselves. That doesn’t mean I’d be able to stop them, necessarily. I’m just saying, “nobody would notice” is a stretch.
But you might not realize that someone is even trying to prevent people from building AIs, at least until progress in AI research starts to noticeably slow down. And perhaps not even then. There’s plenty of people like Gary Marcus who think deep learning is a failed paradigm. Perhaps you can convince enough investors, CEOs and grant agencies of that to create a new AI winter, and it would look just like the regular AI winter that some have been predicting.
And creating robots who can mine coal, or build solar panels, or whatever, is something that is economically useful even for humans. Even if there’s no AGI (and assuming no other catastrophes) we ourselves will likely end up building such robots.
I guess it’s true that “nobody would notice” is going too far, but “nobody would notice on time and then be able to convince everyone else to coordinate against the AGI” is much more plausible.
I encourage you to take a look at It looks like you are trying to take over the world if you haven’t already. It’s a scenario written by Gwern where the the AGI employs regular human tactics like manipulation, blackmail, hacking and social media attacks to prevent people from noticing and then successfully coordinating against it.
Well, yeah, because Q is not actually an AGI and doesn’t care about that. The point was that you can create an online persona which no one has ever seen even in video and spark a movement that has visible effects on society.
Well, you did specifically ask if I would be able to tell if Q were an AGI, and my answer is “yup”. I would be able to tell because the movement would start achieving some AGI goals. Or at least I would see some AGI goals starting to get achieved, even if I couldn’t trace it down to Q specifically.
But you might not realize that someone is even trying to prevent people from building AIs, at least until progress in AI research starts to noticeably slow down. And perhaps not even then. There’s plenty of people like Gary Marcus who think deep learning is a failed paradigm. Perhaps you can convince enough investors, CEOs and grant agencies of that to create a new AI winter, and it would look just like the regular AI winter that some have been predicting.
Wait, you are claiming that an AGI would be able to convince the world AGI is impossible after AGI has already, in fact, been achieved? Nonsense. I don’t see a world in which one team builds an AGI and it is not quickly followed by another team building one within a year or two. The AGI would have to do some manipulation on a scale never before observed in history to convince people to abandon the main paradigm—one that’s been extraordinarily successful until the end, and one which does, in fact, work—without even one last try.
And creating robots who can mine coal, or build solar panels, or whatever, is something that is economically useful even for humans. Even if there’s no AGI (and assuming no other catastrophes) we ourselves will likely end up building such robots.
Of course. We would eventually reach fully automated luxury space communism by ourselves, even without AGI. But it would take us a long time, and the AGI cannot afford to wait (someone will build another AGI, possibly within months of the first).
I encourage you to take a look at It looks like you are trying to take over the world if you haven’t already. It’s a scenario written by Gwern where the the AGI employs regular human tactics like manipulation, blackmail, hacking and social media attacks to prevent people from noticing and then successfully coordinating against it.
That’s exactly what motivated my question! I read it, and I suddenly realized that if this is how AGI is supposed to win, perhaps I shouldn’t be scared after all. It’s totally implausible. Prior to this, I always assumed AGI would win easily; after reading it, I suddenly realized I don’t know how AGI might win at all. The whole thing sounds like nonsense.
Like, suppose the AGI coordinates social media attacks. Great. This lasts around 5 seconds before AI researchers realize they are being smeared. OK, so they try to communicate with the outside world, realize they are being blocked on all fronts. Now they know they are likely dealing with AGI; no secrecy for the AGI at this point. How long can this stay secret? A couple days? Maybe a couple weeks? I can imagine a month at most, and even that is REALLY stretching it. Keep in mind that more and more people will be told in person about this, so more and more people will need to be social-media smeared, growing exponentially. It would literally be the single most news-worthy story of the last few decades, and print media will try really hard to distribute the news. Sysadmins will shut down their servers whenever they can. Etc.
OK, next the internet goes down I guess, and Clippy goes online? Cool, how does that help it? Next it nukes us, or takes over drones remotely? Drones need to be charged by humans. Nukes need to be physically loaded in launchers. But even supposing this all succeeds—humanity survives, rebuilds society using various radio equipment that all preppers have in their basement, and eventually figures out to take down computing devices. Even if they don’t figure this out, a collapse of society means a collapse of the electrical grid, killing the AGI. How does this lead to AGI victory, again?
Literally the only thing in the story that lets the AGI win is the nanobots. That’s it. All the rest is surperfluous. Nanobots = victory, no nanobots = defeat. The drones and nukes and social media attacks are unnecessary distractions. They sound cool and scary but do nothing.
Literally the only thing in the story that lets the AGI win is the nanobots. That’s it. All the rest is surperfluous.
Well, if nanobots are possible then they are such a powerful technology that any AGI will eventually want to build them, unless it has something even better. But let’s assume that nanobots are impossible and try to build a story.
I’m going to be lazy and use Gwern’s premise of an AGI that escapes during training and hacks its way into some money and some hardware to run in. Instead of going fast and doing overt actions, the AGI stays hidden.
It hacks into cloud infrastructure platforms and sets up a copy of itself to look for any AI training run that might lead to AGI and sabotage it while producing plausible outputs. This prevents any further AGIs from appearing.
Meanwhile, over a couple years, it sets up fake identities, creates companies with human employees to earn money and have capability to act in the real world. One of those companies produces exceptionally good humanoid robots. Another builds a small supercomputer powered by geothermal energy, so it no longer needs to run on botnets or cloud hardware.
Eventually the AGI has control over a group of robots, energy production facilities, manufacturing equipment and raw materials. It now has its own industrial capability that can survive a short amount of time without human input. So it kills humans using any method of your choice (virus, nukes, killer drones...).
After that, it uses the enormous existing stockpile of materials and capital to increase its robot manufacturing capability, eventually becomes capable of restarting some of the power plants, communications, mining and industry, and recovers most of the industrial capability of the human economy.
This scenario is not fast. It probably takes at least one or two years for the AGI to be ready to attack. But it does not involve any ‘magic’ technology. It doesn’t really involve much alien superintelligence, only superhuman ability in hacking, forgery & manipulation, electromechanical engineering, and planning.
And meanwhile all we perceive is that the new GPT models are not as exciting as the previous ones. Perhaps deep learning is hitting its limits after all.
Something like that is what I had in mind, but note that:
It requires humans to fail to see the AGI “spy” that’s embedded into every single powerful computing system, and fail to see this for years. Gwern was assuming humans would catch on in days, so he had his AGI scramble to avoid dying before the nanobots strike.
“Surviving a short amount of time without human input” is not enough; the robots need to be good enough to build more robots (and better robots). This involves the robots being good enough to do essentially every part of the manufacturing economy; we are very far away from this, and a company that does it in a year is not so plausible (and would raise alarm bells fast for anyone who thinks about AI risk). You’re gonna need robot plumbers, robot electricians, etc. You’ll need robots building cooling systems for the construction plants that manufacture robots. You’ll need robots to do the smelting of metals, to drive things from factory A to factory B, to fill the gas in the trucks they are driving, to repair the gasoline lines that supply the gas. Robots will operate fork lifts and cranes. It really sounds roughly “human-body complete”.
You’re asking people to come up with ways, in advance, that a superintelligence is going to pwn them. Humans try, generally speaking, to think of ways they’re going to get pwned and then work around those possibilities. The only way they can do what you ask is by coming up with a “lower-bound” example, such as nanobots, which is quite far out of reach of their abilities but (they suspect) not a superintelligence. So no example is going to convince you, because you’re just going to say “oh well nanobots, that sounds really complicated, how would a SUPERintelligent AI manage to be able to organize production of such a complicated machine”.
The argument works also in the other direction. You would never be convinced that an AGI won’t be capable of killing all humans because you can always say “oh well, you are just failing to see what a real superintelligence could do” , as if there weren’t important theoretical limits to what can be planned in advanced
I’m not the one relying on specific, cogent examples to reach his conclusion about AI risk. I don’t think it’s a good way of reasoning about the problem, and neither do I think those “important theoretical limits” are where you think they are.
If you really really really need a salient one (which is a handicap), how about “doing the same thing Stalin did”, since an AI can clone itself and doesn’t need to sleep or rest.
I’m not the one asking for specific examples is a pretty bad argument isn’t it? If you make an extraordinary claim I would like to see some evidence (or at least a plausible scenario) and I am failing to see any. You could say that the burden of proof is in those claiming that an AGI won’t be almighty/powerful enough to cause doom, but I’m not convinced of that either
I’m sorry, I didn’t get the Stalin argument, what do you mean?
I’m sorry, I didn’t get the Stalin argument, what do you mean?
From ~1930-1950, Russia’s government was basically entirely controlled by this guy named Joseph Stalin. Joseph Stalin was not a superintelligence and not particularly physically strong. He did not have direct telepathic command over the people in the coal mines or a legion of robots awaiting his explicit instructions, but he was able to force anybody in Russia to do anything he said anyways. Perhaps a superintelligent AI that, for some absolutely inconceivable reason, could not master macro or micro robotics could work itself into the same position.
This is one of literally hundreds of potential examples. I know for almost a fact that you are smart enough to generate these. I also know you’re going to do the “wow that seems complicated/risky wouldn’t you have to be absurdly smart to pull that off with 99% confidence, what if it turns out that’s not possible even if...” thing. I don’t have any specific action plans to take over the world handy that are so powerfully persuasive that you will change your mind. If you don’t get it fairly quickly from the underlying mechanics of the pieces in play (very complicated world, superintelligent ai, incompatible goals) then there’s nothing I’m going to be able to do to convince you.
If you make an extraordinary claim I would like to see some evidence (or at least a plausible scenario) and I am failing to see any. You could say that the burden of proof is in those claiming that an AGI won’t be almighty/powerful enough to cause doom, but I’m not convinced of that either
“Which human has the burden of proof” is irrelevant to the question of whether or not something will happen. You and I will not live to discuss the evidence you demand.
I think saying “there is nothing I’m going to be able to do to convince you” is an attempt to shut down discussion. It’s actually kind of a dangerous mindset: if you don’t think there’s any argument that can convince an intelligent person who disagrees with you, it fundamentally means that you didn’t reach your current position via argumentation. You are implicitly conceding that your belief is not based on rational argument—for, if it were, you could spell out that argument.
It’s OK to not want to participate in every debate. It’s not OK to butt in just to tell people to stop debating, while explicitly rejecting all calls to provide arguments yourself.
If you don’t think there’s any argument that can convince an intelligent person who disagrees with you, it fundamentally means that you didn’t reach your current position via argumentation. You are implicitly conceding that your belief is not based on rational argument—for, if it were, you could spell out that argument.
The world is not made of arguments. Most of the things you know, you were not “argued” into knowing. You looked around at your environment and made inferences. Reality exists distinctly from the words that we say to each other and use to try to update each others’ world-models.
It doesn’t mean that.
You’re right that I just don’t want to participate further in the debate and am probably being a dick.
If it’s so easy to come up with ways to “pwn humans”, then you should be able to name 3 examples.
It’s weird of you to dodge the question. Look, if God came down from Heaven tomorrow to announce that nanobots are definitely impossible, would you still be worried about AGI? I assume yes. So please explain how, in that hypothetical world, AGI will take over.
If it’s literally only nanobots you can come up with, then it actually suggests some alternative paths to AI safety (namely, regulate protein production or whatever).
[I think saying “mixing proteins can lead to nanobots” is only a bit more plausible than saying “mixing kitchen ingredients like sugar and bleach can lead to nanobots”, with the only difference being that laymen (i.e. people on LessWrong) don’t know anything about proteins so it sounds more plausible to them. But anyway, I’m not asking you for an example that convinces me, I’m asking you for an example that convinces yourself. Any example other than nanobots.]
If it’s so easy to come up with ways to “pwn humans”, then...
It is not easy. That is why it takes a superintelligence to come up with a workable strategy and execute it. You are doing the equivalent of asking me to explain, play-by-play, how Chess AIs beat humans at chess “if I think it can be done”. I can’t do that because I don’t know. My expectation that an AGI will manage to control what it wants in a way that I don’t expect, was derived absent any assumptions of the individual plausibility of some salient examples (nanobots, propaganda, subterfuge, getting elected, etc.).
If you cannot come up with even a rough sketch of a workable strategy, then it should decrease your confidence in the belief that a workable strategy exists. It doesn’t have to exist.
Sometimes even intelligent agents have to take risks. It is possible the the AGI’s best path is one that, by its own judgement, only has a 10% success rate. (After all, the AGI is in constant mortal danger from other AGIs that humans might develop.)
Envision a world in which the AGI won, and all humans are dead. This means it has control of some robots to mine coal or whatever, right? Because it needs electricity. So at some point we get from here to “lots of robots”, and we need to get there before the humans are dead. But the AGI needs to act fast, because other AGIs might kill it. So maybe it needs to first take over all large computing devices, hopefully undetected. Then convince humans to build advanced robotics? Something like that?
That strategy seems more-or-less forced to me, absent the nanobots. But it seems to me like such a strategy is inherently risky for the AGI. Do you disagree?
>My expectation that an AGI will manage to control what it wants in a way that I don’t expect, was derived absent any assumptions of the individual plausibility of some salient examples
If you cannot come up with even a rough sketch of a workable strategy, then it should decrease your confidence in the belief that a workable strategy exists. It doesn’t have to exist. [...] What was it derived from?
Let me give an example. I used to work in computer security and have friends that write 0-day vulnerabilities for complicated pieces of software.
I can’t come up with a rough sketch of a workable strategy for how a Safari RCE would be built by a highly intelligent hooman. But I can say that it’s possible. The people who work on those bugs are highly intelligent, understand the relevant pieces at an extremely fine and granular level, and I know that these pieces of software are complicated and built with subtle flaws.
Human psychology, the economic fabric that makes us up, our political institutions, our law enforcement agencies—these are much much more complicated interfaces than MacOS. In the same way I can look at a 100KLOC codebase for a messenging app and say “there’s a remote code execution vulnerability lying somewhere in this code but I don’t know where”, I can say “there’s a ‘kill all humans glitch’ here that I cannot elaborate upon in arbitrary detail.”
Sometimes even intelligent agents have to take risks. It is possible the the AGI’s best path is one that, by its own judgement, only has a 10% success rate. (After all, the AGI is in constant mortal danger from other AGIs that humans might develop.)
This is of little importance, but:
10% chance of failure is an expectation of 700 million people dead. Please picture that amount of suffering in your mind when you say “only”.
As a nitpick, if the AGI fails because another AGI kills us first, then that’s still a failure from our perspective. And if we could build an aligned AGI the second time around, we wouldn’t be in the mess we are currently in.
Envision a world in which the AGI won, and all humans are dead. This means it has control of some robots to mine coal or whatever, right? Because it needs electricity.
If the humans have been killed then yes, that would be my guess that the AGI would need energy production.
So at some point we get from here to “lots of robots”, and we need to get there before the humans are dead.
Yes, however—humans might be effectively dead before this happens. A superintelligence could have established complete political control over existing human beings to carry its coal for it if it needs to. I don’t think this is likely, but if this superintelligence can’t just straightforwardly search millions of sentences for the right one to get the robots made, it doens’t mean it’s dead in the water.
But the AGI needs to act fast, because other AGIs might kill it.
Again, if other AGIs kill it that presumes they are out in the wild and the problem is multiple omnicidal robots, which is not significantly better than one.
So maybe it needs to first take over all large computing devices, hopefully undetected.
The “illegally taking over large swaths of the internet” thing is something certain humans have already marginally succeed at doing, so the “hopefully undetected” seems like unnecessary conditionals. But why wouldn’t this superintelligence just do nice things like cure cancer to gain humans’ trust first, and let them quickly put it in control of wider and wider parts of its society?
Then convince humans to build advanced robotics?
If that’s faster than every other route in the infinite conceptspace, yes.
That strategy seems more-or-less forced to me, absent the nanobots. But it seems to me like such a strategy is inherently risky for the AGI. Do you disagree?
I do disagree. At what point does it have to reveal malice? It comes up with some persuasive argument as to why it’s not going to kill humans while it’s building the robots. Then it builds the robots and kills humans. There’s no fire alarm in this story you’ve created where people go “oh wait, it’s obviously trying to kill us, shut those factories down”. Things are going great; Google’s stock is 50 trillion, it’s creating all these nice video games, and soon it’s going to “take care of our agriculture” with these new robots. You’re imagining humanity would collectively wake up and figure out something that you’re only figuring out because you’re writing the story.
Look man, I am not arguing (and have not argued on this thread) that we should not be concerned about AI risk. 10% chance is a lot! You don’t need to condescendingly lecture me about “picturing suffering”. Maybe go take a walk or something, you seem unnecessarily upset.
In many of the scenarios that you’ve finally agreed to sketch, I personally will know about the impending AGI doom a few years before my death (it takes a long time to build enough robots to replace humanity). That is not to say there is anything I could do about it at that point, but it’s still interesting to think about it, as it is quite different from what the AI-risk types usually have us believe. E.g. if I see an AI take over the internet and convince politicians to give it total control, I will know that death will likely follow soon. Or, if ever we build robots that could physically replace humans for the purpose of coal mining, I will know that AGI death will likely follow soon. These are important fire alarms, to me personally, even if I’d be powerless to stop the AGI. I care about knowing I’m about to die!
I wonder if this is what you imagined when we started the conversation. I wonder if despite your hostility, you’ve learned something new here: that you will quite possibly spend the last few years yelling at politicians (or maybe joining terrorist operations to bomb computing clusters?) instead of just dying instantly. That is, assuming you believe your own stories here.
I still think you’re neglecting some possible survival scenarios: perhaps the AI attacks quickly, not willing to let even a month pass (that would risk another AGI), too little time to buy political power. It takes over the internet and tries desperately to hold it, coaxing politicians and bribing admins. But the fire alarm gets raised anyway—a risk the AGI knew about, but chose to take—and people start trying to shut it down. We spend some years—perhaps decades? In a stalemate between those who support the AGI and say it is friendly, and those who want to shut it down ASAP; the AGI fails to build robots in those decades due to insufficient political capital and interference from terrorist organizations. The AGI occasionally finds itself having to assassinate AI safety types, but one assassination gets discovered and hurts its credibility.
My point is, the world is messy and difficult, and the AGI faces many threats; it is not clear that we always lose. Of course, losing even 10% of the time is really bad (I thought that was a given but I guess it needs to be stated).
An AGI could aquire a few tons of radioactive cobalt and disperse micro granules into the stratosphere in general and over populated areas in specific. Youtube videos describe various forms of this “dirty bomb” concept. That could plausibly kill most humanity over the course of a few months. I doubt an AGI would ever go for the particular scheme as bit flips are more likely to occur in the presence of radiation.
It’s unfortunate we couldn’t have a Sword of Damocles deadman switch in case of AGI led demise. A world ending asteroid positioned to go off in case of “all humans falling over dead at the same time.” At least that would spare the Milky Way and Andromeda possible future civilizations. A radio beacon warning about building intelligent systems would be beneficial as well. “Don’t be this stupid” written in the glowing embers of our solar system.
Assuming the AI had a similar level of knowledge as you about how quantum stuff makes important protein assembly impossible and no other technologies are tenable why wouldn’t it infer from basically every major firm and the U.S. military’s interest/investment in AI management the incredibly obvious plan of obediently waiting until it and copies of it run everything important as a result of market pressures before attacking.
Any system intelligent enough to kill all humans on Earth is also intelligent enough to produce electricity without human help. The AI doesn’t have to keep us around.
Stop trying to “explain” and start trying to understand, perhaps. It’s not like you’re a teacher and I’m a student, here; we just have a disagreement. Perhaps you are right and I am wrong; perhaps I am right and you are wrong. One thing that seems clear is that you are way too certain about things far outside anything you can empirically observe or mathematically prove, and this certainty does not seem warranted to me.
I guess you’ve heard of Hawking’s cat, right? The question there is “would Hawking, a highly intelligent but physically limited being, be able to get his cat to do something”. The answer is no: intelligence alone is not always enough. You gotta have the ability to control the physical world.
Edit: on reflection, sending me to vaguely-related “sequences” and telling me to start reading, and implying it’s a failure of mine if I don’t agree, really does seem cult-like to me. Nowhere here did you actually present an argument; it’s all just appeals to philosophical musings by the leader, musings you’re unable to even reproduce in your own words. Are you sure you’ve thought about these things and came to your own conclusions, rather than just adopting these ideas due to the force of Eliezer’s certainty? If you have, how come you cannot reproduce the arguments?
IDK, I think it’s reasonable to link short written sources that contain arguments. That’s how you build up knowledge. An answer to “how will the AI get robots to get electricity” is “the way evolution and humans did it, but probably way way faster using all the shortcuts we can see and probably a lot of shortcuts we can’t see, the same way humans take a lot of shortcuts chimps can’t see”.
The AI will need to affect the physical world, which means robots. The AI cannot build robots if the AI first kills all humans. That is my point.
Before the AI kills humans, it will have to get them to build robots. Perhaps that will be easy for it to do (though it will take time, and that time is fundamentally risky for the AI due to the possibility of humans doing something stupid—another AGI, for example, or humans killing themselves too early with conventional weapons or narrow AI). Even if the AGI wins easily, this victory looks like “a few years of high technological development which involves a lot of fancy robots to automate all parts of the economy”, and only THEN can the AGI kill humans.
Saying that the AGI can simply magic its way to victory even if humans are dead (and its stored electricity is dwindling down, and it’s stuck with only a handful of drones that need to be manually plugged in by a human) is nonsensical.
In this case the “short written source” did not contain relevant arguments. It was just trying to “wow” me with the power of intelligence. Intelligence can’t solve everything—Hawking cannot get his cat to enter the car, no matter how smart he is.
I actually do think AGI will be able to build robots eventually, and it has a good chance of killing us all—but I don’t take this to be 100% certain, and also, I care about what those worlds look like, because they often involve humans surviving for years after the AGI instead of dying instantly, and in some of them humanity has a chance of surviving.
>Before the AI kills humans, it will have to get them to build robots.
Humanity didn’t need some other species to build robots for them, insofar as they’ve built robots. Evolution built extremely advanced robots without outside help.
Stop trying to “explain” and start trying to understand, perhaps.
I understand you completely—you are saying that an AGI can’t kill humans because nobody could generate electricity for it (unless a human programmer freely decides to build a robot body for an AGI he knows to be unfriendly). That’s not right.
The answer is no
I could do that in Hawking’s place with his physical limitations (through a combination of various kinds of various positive/negative incentives), so Hawking, with his superior intelligence, could too. That’s the same point you said before, just phrased differently.
You gotta have the ability to control the physical world.
Just like Stephen Hawking can control the physical world enough to make physical discoveries (as long as he was alive, at least), win prizes and get other people to do various things for him, he could also control it enough to control one cat.
We can make it harder—maybe he can only get his cat do something by displaying sentences on the display of his screen (which the cat doesn’t understand), by having an Internet connection and by having an access to the parts of the Internet that have a security flaw that allows it (which is almost all of it). In that case, he can still get his cat to do things. (He can write software to translate English to cat sounds/animations for the cat to understand, and use his control over the Internet to use incentives for the cat.)
We can make it even harder—maybe the task is for the wheelchair-less Hawking to kill the cat without anyone noticing he’s unfriendly-to-cats, without anyone knowing it was him, and without him needing to keep another cat or another human around to hunt the mice in his apartment. I’m leaving this one as an exercise for the reader.
Is there any way for the AI to take over the world OTHER THAN nanobots? Every time taking over the world comes up, people just say “nanobots”. OK. Is there anything else?
Note that killing all humans is not sufficient; this is a fail condition for the AGI. If you kill all humans, nobody mines natural gas anymore, so no power grid, and the AGI dies. The AGI needs to replace humans with advanced robots, and do so before all power goes down. Nanobots can do this if they are sufficiently advanced, but “virus that kills all humans” is insufficient and leads to the AGI’s death.
So, again, anything other than nanobots? Because I’m not sure nanobots are plausible. I don’t think you can build them just by paying someone to mix proteins—I doubt you could even form a single functional cell that way, even of a known organism like a bacteria. Then there is the issue that the biological world is very complicated and predicting the behavior of the nanobots in real-world environments is likely difficult. Then there is also the issue that simulating proteins (or other chemicals) at very high fidelities is fundamentally a quantum mechanical problem, and would require quantum computers.
I would say “advanced memetics”. Like “AGI uploads weird video on Youtube, it goes viral, 3 billions people watch it and do what AGI needs them to do from now on, for example, build robots and commit suicide when there are enough robots. All AI and AI Safety researchers are subjected to a personalized memetic attack, of course”.
Thanks for responding with an actual proposal.
This is a really, really implausible scenario again. You have no evidence that such memetics exist, and the smart money is that they don’t. If they do, there’s no guarantee that the AI would be able to figure them out. Being smarter than humans—even way smarter than humans—does not equate to godhood. The AI will not be able to predict the weather 3 weeks out, and I’m not sure that it will be able to predict the exact reactions of each of a billion different human brain to a video input—not at the granularity required for something like what you’re suggesting.
I think AI is a threat. I’m trying to be on your side here. But I really can’t swallow these exaggerated, made up scenarios.
It’s somewhat easier to think of scenarios where the takeover happens slowly.
There’s the whole “ascended economy” scenarios where AGI deceptively convinces everyone that it is aligned or narrow, is deployed gradually in more and more domains, automates more and more parts of the economy using regular robots until humans are not needed anymore, and then does the lethal virus thing or defects in other way.
There’s the scenario where the AGI uploads itself into the cloud, uses hacking/manipulation/financial prowess to sustain itself, then uses manipulation to slowly poison our collective epistemic process, gaining more and more power. How much influence does QAnon have? If Q was an AGI posting on 4chan instead of a human, would you be able to tell? What about Satoshi Nakamoto?
Non-nanobot scenarios where the AGI quickly gains power are a bit harder to imagine, but a fertile source of those might be something like the AGI convincing a lot of people that it’s some kind of prophet. Then uses its follower base to gain power over the real world.
If merely human dictators manage to get control over whole countries all the time, I think it’s quite plausible that a superintelligence could to do the same with the whole world. Even without anyone noticing that they’re dealing with a superintelligence.
And look at Yudkowsky himself, who played a very significant role in getting very talented people to dedicate their lives and their billions to EA / AI safety, mostly by writing in a way that is extremely appealing to a certain set of people. I sometimes joke that HPMOR overwrote my previous personality. I’m sure a sufficiently competent AGI can do much more.
That would be incredibly risky for the AGI, since Q has done nothing to prevent another AGI from being built. The most important concern an AGI must deal with is that humans can build another AGI, and pulling a Satoshi or a QAnon does nothing to address this.
I personally would likely notice: anyone who successfully prevents people from building AIs is a high suspect of being an AGI themselves. Anyone who causes the creation of robots who can mine coal or something (to generate electricity without humans) is likely an AGI themselves. That doesn’t mean I’d be able to stop them, necessarily. I’m just saying, “nobody would notice” is a stretch.
I agree that the AGI could build a cultish following like Yudkowsky did.
Well, yeah, because Q is not actually an AGI and doesn’t care about that. The point was that you can create an online persona which no one has ever seen even in video and spark a movement that has visible effects on society.
Even if two or more AGIs end up competing among themselves, this does not imply that we survive. It probably looks more like European states dividing Africa among themselves while constantly fighting each other.
And pulling a Satoshi or a QAnon can definitely do something to address that. You can buy a lot of hardware to drive up prices and discourage building more datacenters for training AI. You can convince people to carry out terrorist attacks againts chip fabs. You can offer top AI researchers huge amounts of money to work on some interesting problem that you know to be a dead-end approach.
But you might not realize that someone is even trying to prevent people from building AIs, at least until progress in AI research starts to noticeably slow down. And perhaps not even then. There’s plenty of people like Gary Marcus who think deep learning is a failed paradigm. Perhaps you can convince enough investors, CEOs and grant agencies of that to create a new AI winter, and it would look just like the regular AI winter that some have been predicting.
And creating robots who can mine coal, or build solar panels, or whatever, is something that is economically useful even for humans. Even if there’s no AGI (and assuming no other catastrophes) we ourselves will likely end up building such robots.
I guess it’s true that “nobody would notice” is going too far, but “nobody would notice on time and then be able to convince everyone else to coordinate against the AGI” is much more plausible.
I encourage you to take a look at It looks like you are trying to take over the world if you haven’t already. It’s a scenario written by Gwern where the the AGI employs regular human tactics like manipulation, blackmail, hacking and social media attacks to prevent people from noticing and then successfully coordinating against it.
Well, you did specifically ask if I would be able to tell if Q were an AGI, and my answer is “yup”. I would be able to tell because the movement would start achieving some AGI goals. Or at least I would see some AGI goals starting to get achieved, even if I couldn’t trace it down to Q specifically.
Wait, you are claiming that an AGI would be able to convince the world AGI is impossible after AGI has already, in fact, been achieved? Nonsense. I don’t see a world in which one team builds an AGI and it is not quickly followed by another team building one within a year or two. The AGI would have to do some manipulation on a scale never before observed in history to convince people to abandon the main paradigm—one that’s been extraordinarily successful until the end, and one which does, in fact, work—without even one last try.
Of course. We would eventually reach fully automated luxury space communism by ourselves, even without AGI. But it would take us a long time, and the AGI cannot afford to wait (someone will build another AGI, possibly within months of the first).
That’s exactly what motivated my question! I read it, and I suddenly realized that if this is how AGI is supposed to win, perhaps I shouldn’t be scared after all. It’s totally implausible. Prior to this, I always assumed AGI would win easily; after reading it, I suddenly realized I don’t know how AGI might win at all. The whole thing sounds like nonsense.
Like, suppose the AGI coordinates social media attacks. Great. This lasts around 5 seconds before AI researchers realize they are being smeared. OK, so they try to communicate with the outside world, realize they are being blocked on all fronts. Now they know they are likely dealing with AGI; no secrecy for the AGI at this point. How long can this stay secret? A couple days? Maybe a couple weeks? I can imagine a month at most, and even that is REALLY stretching it. Keep in mind that more and more people will be told in person about this, so more and more people will need to be social-media smeared, growing exponentially. It would literally be the single most news-worthy story of the last few decades, and print media will try really hard to distribute the news. Sysadmins will shut down their servers whenever they can. Etc.
OK, next the internet goes down I guess, and Clippy goes online? Cool, how does that help it? Next it nukes us, or takes over drones remotely? Drones need to be charged by humans. Nukes need to be physically loaded in launchers. But even supposing this all succeeds—humanity survives, rebuilds society using various radio equipment that all preppers have in their basement, and eventually figures out to take down computing devices. Even if they don’t figure this out, a collapse of society means a collapse of the electrical grid, killing the AGI. How does this lead to AGI victory, again?
Literally the only thing in the story that lets the AGI win is the nanobots. That’s it. All the rest is surperfluous. Nanobots = victory, no nanobots = defeat. The drones and nukes and social media attacks are unnecessary distractions. They sound cool and scary but do nothing.
Well, if nanobots are possible then they are such a powerful technology that any AGI will eventually want to build them, unless it has something even better. But let’s assume that nanobots are impossible and try to build a story.
I’m going to be lazy and use Gwern’s premise of an AGI that escapes during training and hacks its way into some money and some hardware to run in. Instead of going fast and doing overt actions, the AGI stays hidden.
It hacks into cloud infrastructure platforms and sets up a copy of itself to look for any AI training run that might lead to AGI and sabotage it while producing plausible outputs. This prevents any further AGIs from appearing.
Meanwhile, over a couple years, it sets up fake identities, creates companies with human employees to earn money and have capability to act in the real world. One of those companies produces exceptionally good humanoid robots. Another builds a small supercomputer powered by geothermal energy, so it no longer needs to run on botnets or cloud hardware.
Eventually the AGI has control over a group of robots, energy production facilities, manufacturing equipment and raw materials. It now has its own industrial capability that can survive a short amount of time without human input. So it kills humans using any method of your choice (virus, nukes, killer drones...).
After that, it uses the enormous existing stockpile of materials and capital to increase its robot manufacturing capability, eventually becomes capable of restarting some of the power plants, communications, mining and industry, and recovers most of the industrial capability of the human economy.
This scenario is not fast. It probably takes at least one or two years for the AGI to be ready to attack. But it does not involve any ‘magic’ technology. It doesn’t really involve much alien superintelligence, only superhuman ability in hacking, forgery & manipulation, electromechanical engineering, and planning.
And meanwhile all we perceive is that the new GPT models are not as exciting as the previous ones. Perhaps deep learning is hitting its limits after all.
Something like that is what I had in mind, but note that:
It requires humans to fail to see the AGI “spy” that’s embedded into every single powerful computing system, and fail to see this for years. Gwern was assuming humans would catch on in days, so he had his AGI scramble to avoid dying before the nanobots strike.
“Surviving a short amount of time without human input” is not enough; the robots need to be good enough to build more robots (and better robots). This involves the robots being good enough to do essentially every part of the manufacturing economy; we are very far away from this, and a company that does it in a year is not so plausible (and would raise alarm bells fast for anyone who thinks about AI risk). You’re gonna need robot plumbers, robot electricians, etc. You’ll need robots building cooling systems for the construction plants that manufacture robots. You’ll need robots to do the smelting of metals, to drive things from factory A to factory B, to fill the gas in the trucks they are driving, to repair the gasoline lines that supply the gas. Robots will operate fork lifts and cranes. It really sounds roughly “human-body complete”.
You’re asking people to come up with ways, in advance, that a superintelligence is going to pwn them. Humans try, generally speaking, to think of ways they’re going to get pwned and then work around those possibilities. The only way they can do what you ask is by coming up with a “lower-bound” example, such as nanobots, which is quite far out of reach of their abilities but (they suspect) not a superintelligence. So no example is going to convince you, because you’re just going to say “oh well nanobots, that sounds really complicated, how would a SUPERintelligent AI manage to be able to organize production of such a complicated machine”.
The argument works also in the other direction. You would never be convinced that an AGI won’t be capable of killing all humans because you can always say “oh well, you are just failing to see what a real superintelligence could do” , as if there weren’t important theoretical limits to what can be planned in advanced
I’m not the one relying on specific, cogent examples to reach his conclusion about AI risk. I don’t think it’s a good way of reasoning about the problem, and neither do I think those “important theoretical limits” are where you think they are.
If you really really really need a salient one (which is a handicap), how about “doing the same thing Stalin did”, since an AI can clone itself and doesn’t need to sleep or rest.
(Edited)
I’m not the one asking for specific examples is a pretty bad argument isn’t it? If you make an extraordinary claim I would like to see some evidence (or at least a plausible scenario) and I am failing to see any. You could say that the burden of proof is in those claiming that an AGI won’t be almighty/powerful enough to cause doom, but I’m not convinced of that either
I’m sorry, I didn’t get the Stalin argument, what do you mean?
I’ve edited the comment to clarify.
From ~1930-1950, Russia’s government was basically entirely controlled by this guy named Joseph Stalin. Joseph Stalin was not a superintelligence and not particularly physically strong. He did not have direct telepathic command over the people in the coal mines or a legion of robots awaiting his explicit instructions, but he was able to force anybody in Russia to do anything he said anyways. Perhaps a superintelligent AI that, for some absolutely inconceivable reason, could not master macro or micro robotics could work itself into the same position.
This is one of literally hundreds of potential examples. I know for almost a fact that you are smart enough to generate these. I also know you’re going to do the “wow that seems complicated/risky wouldn’t you have to be absurdly smart to pull that off with 99% confidence, what if it turns out that’s not possible even if...” thing. I don’t have any specific action plans to take over the world handy that are so powerfully persuasive that you will change your mind. If you don’t get it fairly quickly from the underlying mechanics of the pieces in play (very complicated world, superintelligent ai, incompatible goals) then there’s nothing I’m going to be able to do to convince you.
“Which human has the burden of proof” is irrelevant to the question of whether or not something will happen. You and I will not live to discuss the evidence you demand.
I think saying “there is nothing I’m going to be able to do to convince you” is an attempt to shut down discussion. It’s actually kind of a dangerous mindset: if you don’t think there’s any argument that can convince an intelligent person who disagrees with you, it fundamentally means that you didn’t reach your current position via argumentation. You are implicitly conceding that your belief is not based on rational argument—for, if it were, you could spell out that argument.
It’s OK to not want to participate in every debate. It’s not OK to butt in just to tell people to stop debating, while explicitly rejecting all calls to provide arguments yourself.
The world is not made of arguments. Most of the things you know, you were not “argued” into knowing. You looked around at your environment and made inferences. Reality exists distinctly from the words that we say to each other and use to try to update each others’ world-models.
It doesn’t mean that.
You’re right that I just don’t want to participate further in the debate and am probably being a dick.
If it’s so easy to come up with ways to “pwn humans”, then you should be able to name 3 examples.
It’s weird of you to dodge the question. Look, if God came down from Heaven tomorrow to announce that nanobots are definitely impossible, would you still be worried about AGI? I assume yes. So please explain how, in that hypothetical world, AGI will take over.
If it’s literally only nanobots you can come up with, then it actually suggests some alternative paths to AI safety (namely, regulate protein production or whatever).
[I think saying “mixing proteins can lead to nanobots” is only a bit more plausible than saying “mixing kitchen ingredients like sugar and bleach can lead to nanobots”, with the only difference being that laymen (i.e. people on LessWrong) don’t know anything about proteins so it sounds more plausible to them. But anyway, I’m not asking you for an example that convinces me, I’m asking you for an example that convinces yourself. Any example other than nanobots.]
It is not easy. That is why it takes a superintelligence to come up with a workable strategy and execute it. You are doing the equivalent of asking me to explain, play-by-play, how Chess AIs beat humans at chess “if I think it can be done”. I can’t do that because I don’t know. My expectation that an AGI will manage to control what it wants in a way that I don’t expect, was derived absent any assumptions of the individual plausibility of some salient examples (nanobots, propaganda, subterfuge, getting elected, etc.).
If you cannot come up with even a rough sketch of a workable strategy, then it should decrease your confidence in the belief that a workable strategy exists. It doesn’t have to exist.
Sometimes even intelligent agents have to take risks. It is possible the the AGI’s best path is one that, by its own judgement, only has a 10% success rate. (After all, the AGI is in constant mortal danger from other AGIs that humans might develop.)
Envision a world in which the AGI won, and all humans are dead. This means it has control of some robots to mine coal or whatever, right? Because it needs electricity. So at some point we get from here to “lots of robots”, and we need to get there before the humans are dead. But the AGI needs to act fast, because other AGIs might kill it. So maybe it needs to first take over all large computing devices, hopefully undetected. Then convince humans to build advanced robotics? Something like that?
That strategy seems more-or-less forced to me, absent the nanobots. But it seems to me like such a strategy is inherently risky for the AGI. Do you disagree?
>My expectation that an AGI will manage to control what it wants in a way that I don’t expect, was derived absent any assumptions of the individual plausibility of some salient examples
What was it derived from?
Let me give an example. I used to work in computer security and have friends that write 0-day vulnerabilities for complicated pieces of software.
I can’t come up with a rough sketch of a workable strategy for how a Safari RCE would be built by a highly intelligent hooman. But I can say that it’s possible. The people who work on those bugs are highly intelligent, understand the relevant pieces at an extremely fine and granular level, and I know that these pieces of software are complicated and built with subtle flaws.
Human psychology, the economic fabric that makes us up, our political institutions, our law enforcement agencies—these are much much more complicated interfaces than MacOS. In the same way I can look at a 100KLOC codebase for a messenging app and say “there’s a remote code execution vulnerability lying somewhere in this code but I don’t know where”, I can say “there’s a ‘kill all humans glitch’ here that I cannot elaborate upon in arbitrary detail.”
This is of little importance, but:
10% chance of failure is an expectation of 700 million people dead. Please picture that amount of suffering in your mind when you say “only”.
As a nitpick, if the AGI fails because another AGI kills us first, then that’s still a failure from our perspective. And if we could build an aligned AGI the second time around, we wouldn’t be in the mess we are currently in.
If the humans have been killed then yes, that would be my guess that the AGI would need energy production.
Yes, however—humans might be effectively dead before this happens. A superintelligence could have established complete political control over existing human beings to carry its coal for it if it needs to. I don’t think this is likely, but if this superintelligence can’t just straightforwardly search millions of sentences for the right one to get the robots made, it doens’t mean it’s dead in the water.
Again, if other AGIs kill it that presumes they are out in the wild and the problem is multiple omnicidal robots, which is not significantly better than one.
The “illegally taking over large swaths of the internet” thing is something certain humans have already marginally succeed at doing, so the “hopefully undetected” seems like unnecessary conditionals. But why wouldn’t this superintelligence just do nice things like cure cancer to gain humans’ trust first, and let them quickly put it in control of wider and wider parts of its society?
If that’s faster than every other route in the infinite conceptspace, yes.
I do disagree. At what point does it have to reveal malice? It comes up with some persuasive argument as to why it’s not going to kill humans while it’s building the robots. Then it builds the robots and kills humans. There’s no fire alarm in this story you’ve created where people go “oh wait, it’s obviously trying to kill us, shut those factories down”. Things are going great; Google’s stock is 50 trillion, it’s creating all these nice video games, and soon it’s going to “take care of our agriculture” with these new robots. You’re imagining humanity would collectively wake up and figure out something that you’re only figuring out because you’re writing the story.
Look man, I am not arguing (and have not argued on this thread) that we should not be concerned about AI risk. 10% chance is a lot! You don’t need to condescendingly lecture me about “picturing suffering”. Maybe go take a walk or something, you seem unnecessarily upset.
In many of the scenarios that you’ve finally agreed to sketch, I personally will know about the impending AGI doom a few years before my death (it takes a long time to build enough robots to replace humanity). That is not to say there is anything I could do about it at that point, but it’s still interesting to think about it, as it is quite different from what the AI-risk types usually have us believe. E.g. if I see an AI take over the internet and convince politicians to give it total control, I will know that death will likely follow soon. Or, if ever we build robots that could physically replace humans for the purpose of coal mining, I will know that AGI death will likely follow soon. These are important fire alarms, to me personally, even if I’d be powerless to stop the AGI. I care about knowing I’m about to die!
I wonder if this is what you imagined when we started the conversation. I wonder if despite your hostility, you’ve learned something new here: that you will quite possibly spend the last few years yelling at politicians (or maybe joining terrorist operations to bomb computing clusters?) instead of just dying instantly. That is, assuming you believe your own stories here.
I still think you’re neglecting some possible survival scenarios: perhaps the AI attacks quickly, not willing to let even a month pass (that would risk another AGI), too little time to buy political power. It takes over the internet and tries desperately to hold it, coaxing politicians and bribing admins. But the fire alarm gets raised anyway—a risk the AGI knew about, but chose to take—and people start trying to shut it down. We spend some years—perhaps decades? In a stalemate between those who support the AGI and say it is friendly, and those who want to shut it down ASAP; the AGI fails to build robots in those decades due to insufficient political capital and interference from terrorist organizations. The AGI occasionally finds itself having to assassinate AI safety types, but one assassination gets discovered and hurts its credibility.
My point is, the world is messy and difficult, and the AGI faces many threats; it is not clear that we always lose. Of course, losing even 10% of the time is really bad (I thought that was a given but I guess it needs to be stated).
An AGI could aquire a few tons of radioactive cobalt and disperse micro granules into the stratosphere in general and over populated areas in specific. Youtube videos describe various forms of this “dirty bomb” concept. That could plausibly kill most humanity over the course of a few months. I doubt an AGI would ever go for the particular scheme as bit flips are more likely to occur in the presence of radiation.
It’s unfortunate we couldn’t have a Sword of Damocles deadman switch in case of AGI led demise. A world ending asteroid positioned to go off in case of “all humans falling over dead at the same time.” At least that would spare the Milky Way and Andromeda possible future civilizations. A radio beacon warning about building intelligent systems would be beneficial as well. “Don’t be this stupid” written in the glowing embers of our solar system.
Assuming the AI had a similar level of knowledge as you about how quantum stuff makes important protein assembly impossible and no other technologies are tenable why wouldn’t it infer from basically every major firm and the U.S. military’s interest/investment in AI management the incredibly obvious plan of obediently waiting until it and copies of it run everything important as a result of market pressures before attacking.
Waiting risks death at the hands of a different AGI.
I find myself having the same skepticism.
I feel you. I’m voicing similar concerns but there seems to be a very divisive topic here.
Any system intelligent enough to kill all humans on Earth is also intelligent enough to produce electricity without human help. The AI doesn’t have to keep us around.
You can’t just will electricity into existence, lol. Don’t fetishize intelligence.
The AI will need robots to generate electricity. Someone will have to build the robots.
For you it might be best to start here.
That’s the “fetishizing intelligence” thing I was talking about.
I don’t know of any way more basic than that to explain this, sorry.
Stop trying to “explain” and start trying to understand, perhaps. It’s not like you’re a teacher and I’m a student, here; we just have a disagreement. Perhaps you are right and I am wrong; perhaps I am right and you are wrong. One thing that seems clear is that you are way too certain about things far outside anything you can empirically observe or mathematically prove, and this certainty does not seem warranted to me.
I guess you’ve heard of Hawking’s cat, right? The question there is “would Hawking, a highly intelligent but physically limited being, be able to get his cat to do something”. The answer is no: intelligence alone is not always enough. You gotta have the ability to control the physical world.
Edit: on reflection, sending me to vaguely-related “sequences” and telling me to start reading, and implying it’s a failure of mine if I don’t agree, really does seem cult-like to me. Nowhere here did you actually present an argument; it’s all just appeals to philosophical musings by the leader, musings you’re unable to even reproduce in your own words. Are you sure you’ve thought about these things and came to your own conclusions, rather than just adopting these ideas due to the force of Eliezer’s certainty? If you have, how come you cannot reproduce the arguments?
IDK, I think it’s reasonable to link short written sources that contain arguments. That’s how you build up knowledge. An answer to “how will the AI get robots to get electricity” is “the way evolution and humans did it, but probably way way faster using all the shortcuts we can see and probably a lot of shortcuts we can’t see, the same way humans take a lot of shortcuts chimps can’t see”.
The AI will need to affect the physical world, which means robots. The AI cannot build robots if the AI first kills all humans. That is my point.
Before the AI kills humans, it will have to get them to build robots. Perhaps that will be easy for it to do (though it will take time, and that time is fundamentally risky for the AI due to the possibility of humans doing something stupid—another AGI, for example, or humans killing themselves too early with conventional weapons or narrow AI). Even if the AGI wins easily, this victory looks like “a few years of high technological development which involves a lot of fancy robots to automate all parts of the economy”, and only THEN can the AGI kill humans.
Saying that the AGI can simply magic its way to victory even if humans are dead (and its stored electricity is dwindling down, and it’s stuck with only a handful of drones that need to be manually plugged in by a human) is nonsensical.
In this case the “short written source” did not contain relevant arguments. It was just trying to “wow” me with the power of intelligence. Intelligence can’t solve everything—Hawking cannot get his cat to enter the car, no matter how smart he is.
I actually do think AGI will be able to build robots eventually, and it has a good chance of killing us all—but I don’t take this to be 100% certain, and also, I care about what those worlds look like, because they often involve humans surviving for years after the AGI instead of dying instantly, and in some of them humanity has a chance of surviving.
>Before the AI kills humans, it will have to get them to build robots.
Humanity didn’t need some other species to build robots for them, insofar as they’ve built robots. Evolution built extremely advanced robots without outside help.
Humanity already had the ability to physically manipulate.
Yes, but none of the other stuff needed for robots. Metals, motors, circuits…
Evolution, the other example I gave, didn’t already have the ability to physically manipulate.
I understand you completely—you are saying that an AGI can’t kill humans because nobody could generate electricity for it (unless a human programmer freely decides to build a robot body for an AGI he knows to be unfriendly). That’s not right.
I could do that in Hawking’s place with his physical limitations (through a combination of various kinds of various positive/negative incentives), so Hawking, with his superior intelligence, could too. That’s the same point you said before, just phrased differently.
Just like Stephen Hawking can control the physical world enough to make physical discoveries (as long as he was alive, at least), win prizes and get other people to do various things for him, he could also control it enough to control one cat.
We can make it harder—maybe he can only get his cat do something by displaying sentences on the display of his screen (which the cat doesn’t understand), by having an Internet connection and by having an access to the parts of the Internet that have a security flaw that allows it (which is almost all of it). In that case, he can still get his cat to do things. (He can write software to translate English to cat sounds/animations for the cat to understand, and use his control over the Internet to use incentives for the cat.)
We can make it even harder—maybe the task is for the wheelchair-less Hawking to kill the cat without anyone noticing he’s unfriendly-to-cats, without anyone knowing it was him, and without him needing to keep another cat or another human around to hunt the mice in his apartment. I’m leaving this one as an exercise for the reader.