...to hear that 10% - of fairly general populations which aren’t selected for Singulitarian or even transhumanist views—would endorse a takeoff as fast as ‘within 2 years’ is pretty surprising to me.
In the paper human-level AI was defined as follows:
“Define a ‘high–level machine intelligence’ (HLMI) as one that can carry out most human professions at least as well as a typical human.”
Given that definition it doesn’t seem too surprising to me. I guess I have been less skeptical about this than you...
Fast takeoff / intelligence explosion has always seemed to me to be the most controversial premise, which the most people object to, which most consigned SIAI/MIRI to being viewed like cranks;
What sounds crankish is not that a human level AI might reach a superhuman level within 2 years, but the following. In Yudkowsky’s own words (emphasis mine):
I think that at some point in the development of Artificial Intelligence, we are likely to see a fast, local increase in capability—“AI go FOOM”. Just to be clear on the claim, “fast” means on a timescale of weeks or hours rather than years or decades;
These kind of very extreme views are what I have a real problem with. And just to substantiate “extreme views”, here is Luke Muehlhauser:
It might be developed in a server cluster somewhere, but as soon as you plug a superhuman machine into the internet it will be everywhere moments later.
Given that definition it doesn’t seem too surprising to me. I guess I have been less skeptical about this than you...
I don’t think much of typical humans.
These kind of very extreme views are what I have a real problem with.
I see.
And just to substantiate “extreme views”, here is Luke Muehlhauser:
It might be developed in a server cluster somewhere, but as soon as you plug a superhuman machine into the internet it will be everywhere moments later.
That’s not extreme at all, and also not the same as the EY quote. Have you read any computer security papers? You can literally get people to run programs on their computer as root by offering them pennies! That’s the sort of security environment we operate in. Every botnet with millions of computers is a proof of concept.
That’s not extreme at all, and also not the same as the EY quote. Have you read any computer security papers? You can literally get people to run programs on their computer as root by offering them pennies!
He wrote it will be moments later everywhere. Do you claim that it could take over the Internet within moments?
With proper preparation, yes. To reuse my example: it doesn’t take long to register an Amazon account, offer a high-paying HIT with a binary download which opens up a port on the computer, and within minutes multiple people across the world will have run your trojan (well-paying HITs go very fast & Turkers are geographically diverse, especially if the requester doesn’t set requirements on country*); and then one can begin doing all sorts of other things like fuzzing, SMT solvers to automatically extract vulnerabilities from released patches, building botnets, writing flashworms, etc. At that point, the only real question is how stringently one wishes to define ‘moments later’ and ‘everywhere’.
(Within a few seconds? Hard to see how any plausible AI could copy its entire source code & memories over the existing Internet that fast unless it was for some reason already sitting on something like a gigabit link. Within a few minutes? More reasonable. Is there any real difference in safety? No.)
* IIRC a lot of Turk HITs—like for psychology surveys—specify they want Turkers who are from eg America, to reduce variation or get more relevant answers, so Turk requires a declared country for each worker and lets requesters mandate being from particular countries. That said, there’s a lot of incentive for Turkers to lie—to qualify for more and more high-paying HITs—so even if the AI were for some reason to restrict by country, it’d wind up with a lot of diverse foreign computers anyway.
With proper preparation, yes. To reuse my example: it doesn’t take long to register an Amazon account, offer a high-paying HIT with a binary download which opens up a port on the computer, and within minutes multiple people across the world will have run your trojan (well-paying HITs go very fast & Turkers are geographically diverse, especially if the requester doesn’t set requirements on country*); and then one can begin doing all sorts of other things like fuzzing, SMT solvers to automatically extract vulnerabilities from released patches, building botnets, writing flashworms, etc.
Thanks. Looks like my perception is mainly based on my lack of expertise about security and the resulting inferential distance.
Hard to see how any plausible AI could copy its entire source code & memories over the existing Internet that fast unless it was for some reason already sitting on something like a gigabit link.
Are there good reasons to assume that the first such AI won’t be running on a state of the art supercomputer? Take the movie Avatar. The resources needed to render it were: 4,000 Hewlett-Packard servers with 35,000 processor cores with 104 terabytes of RAM and three petabytes of storage. I suppose that it would have been relatively hard to render it on illegally obtained storage and computational resources?
Do we have any estimates on how quickly a superhuman AI’s storage requirements would grow? CERN produces 30 petabytes of data per year. If an AI undergoing an intelligence explosion requires to store vast amounts of data then it will be much harder for it to copy itself.
The uncertainties involved here still seem to be too big to claim that a superhuman intelligence will be everywhere moments after you connect it to the Internet.
Take the movie Avatar. The resources needed to render it were: 4,000 Hewlett-Packard servers with 35,000 processor cores with 104 terabytes of RAM and three petabytes of storage. I suppose that it would have been relatively hard to render it on illegally obtained storage and computational resources?
I don’t think so. Consider botnets.
How hard is it to buy time on a botnet? Not too hard, since they exist for the sake of selling their services, after all.
Do they have the capacity? Botnets range in size from a few computers to extremes of 30 million computers; if they’re desktops, then average RAM these days tends to be at least 4GB, dual core, and hard drive sizes are >=500GB, briefly looking at the cheapest desktops on Newegg. So to get those specs: 35k cores is 17.5k desktops, for 104tb of RAM you’d need a minimum of 104000 / 4 = 26k computers, and the 3pb would be 6k (3000000 / 500); botnets can’t use 100% of host resources or their attrition will be even higher than usual, so double the numbers, and the minimum of the biggest is 52k. Well within the range of normal botnets (the WP list has 22 botnets which could’ve handled that load). And AFAIK CGI rendering is very parallel, so the botnet being high latency and highly distributed might not be as big an issue as it seems.
How much would it cost? Because botnets are a market, it’s been occasionally studied/reported on by the likes of Brian Krebs (google ‘cost of renting a botnet’). For example, https://www.damballa.com/want-to-rent-an-80-120k-ddos-botnet/ says you could rent a 80-120k botnet for $200 a day, or a 12k botnet for $500 a month—so presumably 5 such botnets would cost a quite reasonable $2500 per month.
Do we have any estimates on how quickly a superhuman AI’s storage requirements would grow? CERN produces 30 petabytes of data per year. If an AI undergoing an intelligence explosion requires to store vast amounts of data then it will be much harder for it to copy itself.
I don’t know. Humans get by adding only a few bits per second to long-term memory, Landauer estimated, but I’m not sure how well that maps onto an AI.
The uncertainties involved here still seem to be too big to claim that a superhuman intelligence will be everywhere moments after you connect it to the Internet.
It may not be able to move itself instantly, but given everything we know about botnets and computer security, it would be able to control a lot of remote computers quickly if prepared, and that opens up a lot of avenues one would rather not have to deal with. (Original AI connects to Internet, seizes a few computers across the world, installs a small seed as root with a carefully packaged set of instructions, small database of key facts around the world, and a bunch of Bitcoin private keys for funding while the seed grows to something approaching the original, and lets itself be wiped by its vigilant overseers.)
It will want to—for safety, to avoid being vulnerable due to having a single point of failure.
However the “take over in moments” refers to just control, I think. Getting the AI to become fully distributed and able to tolerate losing large chunks of hardware will take a fair amount of time and, likely, hardware upgrades.
The two quotes you gave say two pretty different things. What Yudkowsky said about the time-scale of self improvement being weeks or hours, is controversial. FWIW, I think he’s probably right, but I wouldn’t be shocked if it turned out otherwise.
What Luke said was about what happens when an already-superhuman AI gets an Internet connection. This should not be controversial at all. This is merely claiming that a “superhuman machine” is capable of doing something that regular humans already do on a fairly routine basis. The opposite claim—that the AI will not spread to everywhere on the Internet—requires us to believe that there will be a significant shift away from the status quo in computer security. Which is certainly possible, but believing the status quo will hold isn’t an extreme view.
The two quotes you gave say two pretty different things. What Yudkowsky said about the time-scale of self improvement being weeks or hours, is controversial.
My problem with Luke’s quote was the “moments later” part.
I took that as hyperbole. If I were meant to take it literally, then yes, I’d object—but I have no trouble believing that a superintelligent AI would be out of there in a matter of hours to minutes, modulo bandwidth limits, which is ‘instant’ enough for my purposes. Humans suck at computer security.
Yes, applying the SI definition of a moment as 1/2π seconds and the ANSI upper bound of a plural before you must change units, we can derive that he was either claiming world takeover in less than 10/(2π)^1/2 ≈ 3.9894 seconds, or speaking somewhat loosely.
Right, and I’m saying: the “moments later” part of what Luke said is not something that should be surprising or controversial, given the premises. It does not require any thinking that can’t be done in advance, which means the only limiting input is bandwidth, which is both plentiful and becoming more plentiful every year.
Right, and I’m saying: the “moments later” part of what Luke said is not something that should be surprising or controversial, given the premises.
The premise was a superhuman intelligence? I don’t see how it could create a large enough botnet, or find enough exploits, in order to be everywhere moments later. Sounds like magic to me (mind you, a complete layman).
If I approximate “superintelligence” as NSA, then I don’t see how the NSA could have a trojan everywhere moments after the POTUS asked them to take over the Internet. Now I could go further and imagine the POTUS asking the NSA to take it over within 10 years in order to account for the subjective speed with which a superintelligence might think. But I strongly doubt that such a speed could make up for the data the NSA already possess and which the superintelligence still needs to acquire. It also does not make up for the thousands of drones (humans in meatspace) that the NSA controls. And since the NSA can’t take over the Internet within moments I believe it is very extreme to claim that a superintelligence can. Though it might be possible within days.
I hope you don’t see this as an attack. I honestly don’t see how that could be possible.
This is not magic, I am not a layman, and your beliefs about computer security are wildly misinformed. Putting trojans on large fractions of the computers on the internet is currently within the reach of, and is actually done by, petty criminals acting alone. While this does involve a fair amount of thinking time, all of this thinking goes into advance preparation, which could be done while still in an AI-box or in advance of an order.
This is not magic, I am not a layman, and your beliefs about computer security are wildly misinformed. Putting trojans on large fractions of the computers on the internet is currently within the reach of, and is actually done by, petty criminals acting alone.
Within moments? I don’t take your word for this, sorry. The only possibility that comes to my mind is by somehow hacking the Windows update servers and then somehow forcefully install new “updates” without user permission.
While this does involve a fair amount of thinking time, all of this thinking goes into advance preparation, which could be done while still in an AI-box or in advance of an order.
So if I uploaded you onto some alien computer, and you had a billion years of subjective time to think about it, then within moments after you got an “Internet” connection you could put a trojan on most computers of that alien society? How would you e.g. figure out zero day exploits of software that you don’t even know exists?
Well, what’s going to slow it down? If you have a backdoor or an exploit, to take over a computer requires a few milliseconds for communications latency and a few milliseconds to run the code to execute the takeover. At this point the new zombie becomes a vector for further infection, you have exponential growth and BOOM!
The only possibility that comes to my mind is by somehow hacking the Windows update servers and then somehow forcefully install new “updates” without user permission.
Wouldn’t have to be Windows; any popular software package with live updates would do, like Acrobat or Java or any major antivirus package. Or you could find a vulnerability that allows arbitrary code execution in any popular push notification service; find one in Apache or a comparably popular Web service, then corrupt all the servers you can find; exploit one in a popular browser, if you can suborn something like Google or Amazon’s front page… there’s lots of stuff you could do. If you have hours instead of moments, phishing attacks and the like become practical, and things get even more fun.
How would you e.g. figure out zero day exploits of software that you don’t even know exists?
Well, presumably you’re running in an environment that has some nontrivial fraction of that software floating around, or at least has access to repos with it. And there’s always fuzzing.
I don’t see how the NSA could have a trojan everywhere moments after the POTUS asked them to take over the Internet.
Given a backdoor or an appropriate zero-day exploit, I would estimate that it would take no longer than a few minutes to gain control over most of the computers connected to the ’net if you’re not worried about detection. It’s not hard. Random people routinely build large botnets without any superhuman abilities.
Most computers are not directly connected to the internet.
Assuming we are not talking about computers in cars and factory control systems, this is a pretty meaningless statement. Yes, most computers sit behind routers, but then basically all computers on the ’net sit behind routers, no one is “directly” connected.
Besides, routers are computers and can be taken over as well.
But not by that zero-day microsoft exploit you found. If your router is a cisco system, you need a cisco zero-day exploit to access the machines behind it, or some other way of bypassing the firewall. Sure, it could take over all the already vulnerable computers, the same ones which are already compromised by botnets. But I object to calling these “most of the computers connected to the ’net”.
But not by that zero-day microsoft exploit you found
The original scenario discussed was the NSA taking over the internet. I assume that the NSA has an extensive collection of backdoors and exploits (cf. Snowden) for Microsoft and Linux and Cisco, etc.
Yes well I thin XiXiDu did himself a disfavor there. If Snowden is to be believed and as various state-sponsored botnets (Stuxnet, Flame, BadBIOS(?)) have shown, the NSA has already “taken over” the internet. They may not have root access on any arbitrary internet-connected machine, but they could get it if they wanted.
My objection (and his?) is against the claim that an AI could replicate this capability in “moments,” according to the “because superhuman!” line of reasoning. I find that bogus.
My objection (and his?) is against the claim that an AI could replicate this capability in “moments,” according to the “because superhuman!” line of reasoning. I find that bogus.
An AI probably wouldn’t need to decompile anything—given the kind of optimizations that one could apply, there’s no particularly strong reason to think one would be any less comfortable in native machine code or, say, Java bytecode than in source. The only reason we are is that it’s closer to natural language and we’re bad at keeping track of a lot of disaggregated state.
That is a monumentally difficult undertaking, unfeasible with current hardware limitations
I think you underestimate the state of the art, such as the SAT/SMT-solver revolution in computer security. They automatically find exploits all the time, against OSes and libraries and APIs.
I think you underestimate the state of the art, such as the SAT/SMT-solver revolution in computer security. They automatically find exploits all the time, against OSes and libraries and APIs.
I think you miss my point. These SAT solvers are extremely expensive, and don’t scale well to large code bases. You can look to the literature to see the state of the art: using large clusters for long-running analysis on small code bases or isolated sections of a library. They do not and cannot with available resources scale up to large scale analysis of an entire OS or network stack … if they did, we humans would have done that already.
So to be clear, this UFAI breakout scenario is assuming the AI already has access to massive amounts of computing hardware, which it can re-purpose to long-duration HPC applications while escaping detection. And even if you find that realistic, I still wouldn’t use the word “momentarily.”
I think you miss my point. These SAT solvers are extremely expensive, and don’t scale well to large code bases. You can look to the literature to see the state of the art: using large clusters for long-running analysis on small code bases or isolated sections of a library. They do not and cannot with available resources scale up to large scale analysis of an entire OS or network stack … if they did, we humans would have done that already.
They have done that already. For example, this paper: “We implement our approach using a popular graph database and demonstrate its efficacy by identifying 18 previously unknown vulnerabilities in the source code of the Linux kernel.”
You can look to the literature to see the state of the art: using large clusters for long-running analysis on small code bases or isolated sections of a library.
Large clusters like… the ones that an AI would be running on?
They do not and cannot with available resources scale up to large scale analysis of an entire OS or network stack … if they did, we humans would have done that already.
They don’t have to scale although that may be possible given increases in computing power (you only need to find an exploit somewhere, not all exploits everywhere), and I am skeptical we humans would, in fact, ‘have done that already’. That claim seems to prove way too much: are existing static code analysis tools applied everywhere? Are existing fuzzers applied everywhere?
It requires an infeasible amount of computation for us humans to do.
Um. Humans—in real life—do run security audits of software. It’s nothing rare or unusual. Frequently these audits are assisted by automated tools (e.g. checking for buffer overruns, etc.). Again, this is happening right now in real life and there are no “infeasible amount of computation” problems.
Doing an audit to catch all vulnerabilities is monstrously hard. But finding some vulnerabilities is a perfectly straightforward technical problem.
It happens routinely that people develop new and improved vulnerability detectors that can quickly find vulnerabilities in existing codebases. I would be unsurprised if better optimization engines in general lead to better vulnerability detectors.
It might be developed in a server cluster somewhere, but as soon as you plug a superhuman machine into the internet it will be everywhere moments later.
It’s not like it’s that hard to hack into servers and run your own computations on them through the internet. Assuming the superintelligence knows enough about the internet to design something like this beforehand (likely since it runs in a server cluster), it seems like the limiting factor here would be bandwidth.
I imagine a highly intelligent human trapped in this sort of situation, with similar prior knowledge and resource access, could build a botnet in a few months. Working on it at full-capacity, non-stop, could bring this down to a few weeks, and it seems plausible to me that with increased intelligence and processing speed, it could build one in a few moments. And of course with access to its own source code, it would be trivial to have it run more copies of itself on the botnet.
Even if you disagree with this line of reasoning, I don’t think it’s fair to paint it as “*very extreme”.
It might be developed in a server cluster somewhere, but as soon as you plug a superhuman machine into the internet it will be everywhere moments later.
Even if you disagree with this line of reasoning, I don’t think it’s fair to paint it as “*very extreme”.
With “very extreme” I was referring to the part where he claims that this will happen “moments later”.
In the paper human-level AI was defined as follows:
Given that definition it doesn’t seem too surprising to me. I guess I have been less skeptical about this than you...
What sounds crankish is not that a human level AI might reach a superhuman level within 2 years, but the following. In Yudkowsky’s own words (emphasis mine):
These kind of very extreme views are what I have a real problem with. And just to substantiate “extreme views”, here is Luke Muehlhauser:
I don’t think much of typical humans.
I see.
That’s not extreme at all, and also not the same as the EY quote. Have you read any computer security papers? You can literally get people to run programs on their computer as root by offering them pennies! That’s the sort of security environment we operate in. Every botnet with millions of computers is a proof of concept.
He wrote it will be moments later everywhere. Do you claim that it could take over the Internet within moments?
With proper preparation, yes. To reuse my example: it doesn’t take long to register an Amazon account, offer a high-paying HIT with a binary download which opens up a port on the computer, and within minutes multiple people across the world will have run your trojan (well-paying HITs go very fast & Turkers are geographically diverse, especially if the requester doesn’t set requirements on country*); and then one can begin doing all sorts of other things like fuzzing, SMT solvers to automatically extract vulnerabilities from released patches, building botnets, writing flashworms, etc. At that point, the only real question is how stringently one wishes to define ‘moments later’ and ‘everywhere’.
(Within a few seconds? Hard to see how any plausible AI could copy its entire source code & memories over the existing Internet that fast unless it was for some reason already sitting on something like a gigabit link. Within a few minutes? More reasonable. Is there any real difference in safety? No.)
* IIRC a lot of Turk HITs—like for psychology surveys—specify they want Turkers who are from eg America, to reduce variation or get more relevant answers, so Turk requires a declared country for each worker and lets requesters mandate being from particular countries. That said, there’s a lot of incentive for Turkers to lie—to qualify for more and more high-paying HITs—so even if the AI were for some reason to restrict by country, it’d wind up with a lot of diverse foreign computers anyway.
Thanks. Looks like my perception is mainly based on my lack of expertise about security and the resulting inferential distance.
Are there good reasons to assume that the first such AI won’t be running on a state of the art supercomputer? Take the movie Avatar. The resources needed to render it were: 4,000 Hewlett-Packard servers with 35,000 processor cores with 104 terabytes of RAM and three petabytes of storage. I suppose that it would have been relatively hard to render it on illegally obtained storage and computational resources?
Do we have any estimates on how quickly a superhuman AI’s storage requirements would grow? CERN produces 30 petabytes of data per year. If an AI undergoing an intelligence explosion requires to store vast amounts of data then it will be much harder for it to copy itself.
The uncertainties involved here still seem to be too big to claim that a superhuman intelligence will be everywhere moments after you connect it to the Internet.
I don’t think so. Consider botnets.
How hard is it to buy time on a botnet? Not too hard, since they exist for the sake of selling their services, after all.
Do they have the capacity? Botnets range in size from a few computers to extremes of 30 million computers; if they’re desktops, then average RAM these days tends to be at least 4GB, dual core, and hard drive sizes are >=500GB, briefly looking at the cheapest desktops on Newegg. So to get those specs: 35k cores is 17.5k desktops, for 104tb of RAM you’d need a minimum of
104000 / 4 = 26kcomputers, and the 3pb would be 6k (3000000 / 500); botnets can’t use 100% of host resources or their attrition will be even higher than usual, so double the numbers, and the minimum of the biggest is 52k. Well within the range of normal botnets (the WP list has 22 botnets which could’ve handled that load). And AFAIK CGI rendering is very parallel, so the botnet being high latency and highly distributed might not be as big an issue as it seems.How much would it cost? Because botnets are a market, it’s been occasionally studied/reported on by the likes of Brian Krebs (google ‘cost of renting a botnet’). For example, https://www.damballa.com/want-to-rent-an-80-120k-ddos-botnet/ says you could rent a 80-120k botnet for $200 a day, or a 12k botnet for $500 a month—so presumably 5 such botnets would cost a quite reasonable $2500 per month.
(That’s much cheaper than Amazon AWS, looks like. https://calculator.s3.amazonaws.com/index.html 17500 t2.medium instances would cost ~$666k a month.)
I don’t know. Humans get by adding only a few bits per second to long-term memory, Landauer estimated, but I’m not sure how well that maps onto an AI.
It may not be able to move itself instantly, but given everything we know about botnets and computer security, it would be able to control a lot of remote computers quickly if prepared, and that opens up a lot of avenues one would rather not have to deal with. (Original AI connects to Internet, seizes a few computers across the world, installs a small seed as root with a carefully packaged set of instructions, small database of key facts around the world, and a bunch of Bitcoin private keys for funding while the seed grows to something approaching the original, and lets itself be wiped by its vigilant overseers.)
Why does the AI have to transfer its source code? I assumed we were just talking about taking over machines as effectors.
It will want to—for safety, to avoid being vulnerable due to having a single point of failure.
However the “take over in moments” refers to just control, I think. Getting the AI to become fully distributed and able to tolerate losing large chunks of hardware will take a fair amount of time and, likely, hardware upgrades.
The two quotes you gave say two pretty different things. What Yudkowsky said about the time-scale of self improvement being weeks or hours, is controversial. FWIW, I think he’s probably right, but I wouldn’t be shocked if it turned out otherwise.
What Luke said was about what happens when an already-superhuman AI gets an Internet connection. This should not be controversial at all. This is merely claiming that a “superhuman machine” is capable of doing something that regular humans already do on a fairly routine basis. The opposite claim—that the AI will not spread to everywhere on the Internet—requires us to believe that there will be a significant shift away from the status quo in computer security. Which is certainly possible, but believing the status quo will hold isn’t an extreme view.
My problem with Luke’s quote was the “moments later” part.
I took that as hyperbole. If I were meant to take it literally, then yes, I’d object—but I have no trouble believing that a superintelligent AI would be out of there in a matter of hours to minutes, modulo bandwidth limits, which is ‘instant’ enough for my purposes. Humans suck at computer security.
Yes, applying the SI definition of a moment as 1/2π seconds and the ANSI upper bound of a plural before you must change units, we can derive that he was either claiming world takeover in less than 10/(2π)^1/2 ≈ 3.9894 seconds, or speaking somewhat loosely.
Hmmmmmmmmm.
Right, and I’m saying: the “moments later” part of what Luke said is not something that should be surprising or controversial, given the premises. It does not require any thinking that can’t be done in advance, which means the only limiting input is bandwidth, which is both plentiful and becoming more plentiful every year.
The premise was a superhuman intelligence? I don’t see how it could create a large enough botnet, or find enough exploits, in order to be everywhere moments later. Sounds like magic to me (mind you, a complete layman).
If I approximate “superintelligence” as NSA, then I don’t see how the NSA could have a trojan everywhere moments after the POTUS asked them to take over the Internet. Now I could go further and imagine the POTUS asking the NSA to take it over within 10 years in order to account for the subjective speed with which a superintelligence might think. But I strongly doubt that such a speed could make up for the data the NSA already possess and which the superintelligence still needs to acquire. It also does not make up for the thousands of drones (humans in meatspace) that the NSA controls. And since the NSA can’t take over the Internet within moments I believe it is very extreme to claim that a superintelligence can. Though it might be possible within days.
I hope you don’t see this as an attack. I honestly don’t see how that could be possible.
This is not magic, I am not a layman, and your beliefs about computer security are wildly misinformed. Putting trojans on large fractions of the computers on the internet is currently within the reach of, and is actually done by, petty criminals acting alone. While this does involve a fair amount of thinking time, all of this thinking goes into advance preparation, which could be done while still in an AI-box or in advance of an order.
Within moments? I don’t take your word for this, sorry. The only possibility that comes to my mind is by somehow hacking the Windows update servers and then somehow forcefully install new “updates” without user permission.
So if I uploaded you onto some alien computer, and you had a billion years of subjective time to think about it, then within moments after you got an “Internet” connection you could put a trojan on most computers of that alien society? How would you e.g. figure out zero day exploits of software that you don’t even know exists?
Well, what’s going to slow it down? If you have a backdoor or an exploit, to take over a computer requires a few milliseconds for communications latency and a few milliseconds to run the code to execute the takeover. At this point the new zombie becomes a vector for further infection, you have exponential growth and BOOM!
Wouldn’t have to be Windows; any popular software package with live updates would do, like Acrobat or Java or any major antivirus package. Or you could find a vulnerability that allows arbitrary code execution in any popular push notification service; find one in Apache or a comparably popular Web service, then corrupt all the servers you can find; exploit one in a popular browser, if you can suborn something like Google or Amazon’s front page… there’s lots of stuff you could do. If you have hours instead of moments, phishing attacks and the like become practical, and things get even more fun.
Well, presumably you’re running in an environment that has some nontrivial fraction of that software floating around, or at least has access to repos with it. And there’s always fuzzing.
Also, nowadays if you can suborn the cell towers taking over all the smartphones becomes fast and easy.
When you are a layman talking to experts, you should actually listen. Don’t make us feel like we’re wasting our time.
Care to address his valid response point in the 2nd paragraph?
Nornagest already answered it; the sets of software in and outside the box aren’t disjoint.
Given a backdoor or an appropriate zero-day exploit, I would estimate that it would take no longer than a few minutes to gain control over most of the computers connected to the ’net if you’re not worried about detection. It’s not hard. Random people routinely build large botnets without any superhuman abilities.
Most computers are not directly connected to the internet.
Assuming we are not talking about computers in cars and factory control systems, this is a pretty meaningless statement. Yes, most computers sit behind routers, but then basically all computers on the ’net sit behind routers, no one is “directly” connected.
Besides, routers are computers and can be taken over as well.
But not by that zero-day microsoft exploit you found. If your router is a cisco system, you need a cisco zero-day exploit to access the machines behind it, or some other way of bypassing the firewall. Sure, it could take over all the already vulnerable computers, the same ones which are already compromised by botnets. But I object to calling these “most of the computers connected to the ’net”.
The original scenario discussed was the NSA taking over the internet. I assume that the NSA has an extensive collection of backdoors and exploits (cf. Snowden) for Microsoft and Linux and Cisco, etc.
Yes well I thin XiXiDu did himself a disfavor there. If Snowden is to be believed and as various state-sponsored botnets (Stuxnet, Flame, BadBIOS(?)) have shown, the NSA has already “taken over” the internet. They may not have root access on any arbitrary internet-connected machine, but they could get it if they wanted.
My objection (and his?) is against the claim that an AI could replicate this capability in “moments,” according to the “because superhuman!” line of reasoning. I find that bogus.
Let me suggest a way:
(1) Gain control of a single machine
(2) Decompile the OS code
(3) Run a security audit on the OS, find exploits
Even easier if the OS is open-sourced.
An AI probably wouldn’t need to decompile anything—given the kind of optimizations that one could apply, there’s no particularly strong reason to think one would be any less comfortable in native machine code or, say, Java bytecode than in source. The only reason we are is that it’s closer to natural language and we’re bad at keeping track of a lot of disaggregated state.
That is a monumentally difficult undertaking, unfeasible with current hardware limitations, certainly impossible in the “moments” timescale.
I think you underestimate the state of the art, such as the SAT/SMT-solver revolution in computer security. They automatically find exploits all the time, against OSes and libraries and APIs.
I think you miss my point. These SAT solvers are extremely expensive, and don’t scale well to large code bases. You can look to the literature to see the state of the art: using large clusters for long-running analysis on small code bases or isolated sections of a library. They do not and cannot with available resources scale up to large scale analysis of an entire OS or network stack … if they did, we humans would have done that already.
So to be clear, this UFAI breakout scenario is assuming the AI already has access to massive amounts of computing hardware, which it can re-purpose to long-duration HPC applications while escaping detection. And even if you find that realistic, I still wouldn’t use the word “momentarily.”
They have done that already. For example, this paper: “We implement our approach using a popular graph database and demonstrate its efficacy by identifying 18 previously unknown vulnerabilities in the source code of the Linux kernel.”
Large clusters like… the ones that an AI would be running on?
They don’t have to scale although that may be possible given increases in computing power (you only need to find an exploit somewhere, not all exploits everywhere), and I am skeptical we humans would, in fact, ‘have done that already’. That claim seems to prove way too much: are existing static code analysis tools applied everywhere? Are existing fuzzers applied everywhere?
Why in the world would a security audit of a bunch of code be “monumentally difficult” for an AI..?
It requires an infeasible amount of computation for us humans to do. Why do you suppose it would be different for an AI?
Um. Humans—in real life—do run security audits of software. It’s nothing rare or unusual. Frequently these audits are assisted by automated tools (e.g. checking for buffer overruns, etc.). Again, this is happening right now in real life and there are no “infeasible amount of computation” problems.
Doing an audit to catch all vulnerabilities is monstrously hard. But finding some vulnerabilities is a perfectly straightforward technical problem.
It happens routinely that people develop new and improved vulnerability detectors that can quickly find vulnerabilities in existing codebases. I would be unsurprised if better optimization engines in general lead to better vulnerability detectors.
It’s not like it’s that hard to hack into servers and run your own computations on them through the internet. Assuming the superintelligence knows enough about the internet to design something like this beforehand (likely since it runs in a server cluster), it seems like the limiting factor here would be bandwidth.
I imagine a highly intelligent human trapped in this sort of situation, with similar prior knowledge and resource access, could build a botnet in a few months. Working on it at full-capacity, non-stop, could bring this down to a few weeks, and it seems plausible to me that with increased intelligence and processing speed, it could build one in a few moments. And of course with access to its own source code, it would be trivial to have it run more copies of itself on the botnet.
Even if you disagree with this line of reasoning, I don’t think it’s fair to paint it as “*very extreme”.
With “very extreme” I was referring to the part where he claims that this will happen “moments later”.
Yes, that was clear. My point is that it isn’t extreme under the mild assumption that the AI has prepared for such an event beforehand.