Slightly superhuman AGIs (that are very productive and have serial speed advantage, but not qualitatively more intelligent) are not directly uncontainably dangerous, if they don’t get an opportunity to build improved versions. The current mode of scaling seems unlikely to suddenly jump from weak systems to superintelligent systems without visibly passing through this stage, if AGIs at this stage don’t immediately escape and improve themselves outside the standard process.
The “if”s seem like possibly big “if”s. If it’s slightly superhuman at computer security, and it has an opportunity to run arbitrary code on some internet-connected machine, then it seems likely that some instance of it, at some point, will find a vulnerability in OpenSSL or something that lets it take over millions of machines on the internet, which it can use to run thousands of instances of itself to brute-force finding more vulnerabilities, and thus bootstrap to taking over as much of the internet as it wants. Which would then let it work on self-improvement and other goals.
A slight advantage in doing computer security research won’t give an entity the ability to take over the internet, by a long shot, especially if it does not have backing by nation state actors. The NSA for instance, as an organisation, has been good at hacking for a long time, and while certainly they can and have done lots of interesting things, they wouldn’t be able to take over the world, probably even if they tried and did it with the backing of the full force of the US military.
Indeed, for some computer security problems, even superintelligence might not confer any advantage at all! It’s perfectly possible, say, that a superintelligence running on a Matrioshka brain a million years hence will find only modest improvements upon current best attacks against the full AES-128. Intelligence allows one to do math better and, occasionally, to find ways and means that side-step mathematical guarantees, but it does not render the adversary omnipotent; an ASI still has to accept (or negotiate around) physical, mathematical and organizational limits to what it can do. In that sense, a lot of the ASI safety debate I think runs on overpowered adversaries, which will in the long run be bad both in terms of achieving ASI safety (because in an overpowered adversary model, real dangers risk remaining unidentified and unfixed) and in terms of realizing the potential benefits of creating AGI/ASI.
I’m not sure how much of this you already know, but the majority of security vulnerabilities are things like “failure to check the bounds of a buffer” or “failure to sanitize or escape user input before plugging it into a command parser”—dumb mistakes in implementation, in other words. It’s much rarer to find a problem in the cryptographic algorithms (although that happens occasionally, like MD5). If we look through OpenSSL’s already-fixed vulnerabilities list:
Browser page search says that “buffer over” (as in buffer overflow, overrun, or over-read) appears on the page 24 times (although there’s double-counting there, as it tends to appear in the title and once or twice in the description). You don’t need to be a world-class security researcher to find these security holes; it’s more a matter of (a) happening to look in the right place, (b) having some knowledge and creativity and intelligence in figuring out how it could be exploited.
[the NSA] wouldn’t be able to take over the world
The NSA isn’t able to run thousands of copies of itself on machines it hacks into, nor use that to quickly create more powerful instances of itself. So that part of world domination is clearly out.
But if you mean “the NSA wouldn’t be able to do that first step of taking over millions of computers” (which the hypothetical AI would then use to bootstrap)… I disagree. Do you know about Stuxnet? Believed to be written by some combination of the NSA and Mossad. Observe:
Stuxnet attacked Windows systems using an unprecedented four zero-day attacks (plus the CPLINKvulnerability and a vulnerability used by the Conficker worm[63]). It is initially spread using infected removable drives such as USB flash drives,[21][45] which contain Windows shortcut files to initiate executable code.[64] The worm then uses other exploits and techniques such as peer-to-peerremote procedure call (RPC) to infect and update other computers inside private networks that are not directly connected to the Internet.[65][66][67] The number of zero-day exploits used is unusual, as they are highly valued and malware creators do not typically make use of (and thus simultaneously make visible) four different zero-day exploits in the same worm.[23] Amongst these exploits were remote code execution on a computer with Printer Sharing enabled,[68] and the LNK/PIF vulnerability,[69] in which file execution is accomplished when an icon is viewed in Windows Explorer, negating the need for user interaction.[70]
So the NSA and/or Mossad had all these exploits and were sitting on them. Makes it plausible that, today, they have more that they’re sitting on.
The Conficker worm itself apparently did take over an estimated “9 million to 15 million” machines. So clearly that is doable.
The “if”s seem like possibly big “if”s. If it’s slightly superhuman at computer security, and it has an opportunity to run arbitrary code on some internet-connected machine, then it seems likely that some instance of it, at some point, will find a vulnerability in OpenSSL or something that lets it take over millions of machines on the internet, which it can use to run thousands of instances of itself to brute-force finding more vulnerabilities, and thus bootstrap to taking over as much of the internet as it wants. Which would then let it work on self-improvement and other goals.
A slight advantage in doing computer security research won’t give an entity the ability to take over the internet, by a long shot, especially if it does not have backing by nation state actors. The NSA for instance, as an organisation, has been good at hacking for a long time, and while certainly they can and have done lots of interesting things, they wouldn’t be able to take over the world, probably even if they tried and did it with the backing of the full force of the US military.
Indeed, for some computer security problems, even superintelligence might not confer any advantage at all! It’s perfectly possible, say, that a superintelligence running on a Matrioshka brain a million years hence will find only modest improvements upon current best attacks against the full AES-128. Intelligence allows one to do math better and, occasionally, to find ways and means that side-step mathematical guarantees, but it does not render the adversary omnipotent; an ASI still has to accept (or negotiate around) physical, mathematical and organizational limits to what it can do. In that sense, a lot of the ASI safety debate I think runs on overpowered adversaries, which will in the long run be bad both in terms of achieving ASI safety (because in an overpowered adversary model, real dangers risk remaining unidentified and unfixed) and in terms of realizing the potential benefits of creating AGI/ASI.
I’m not sure how much of this you already know, but the majority of security vulnerabilities are things like “failure to check the bounds of a buffer” or “failure to sanitize or escape user input before plugging it into a command parser”—dumb mistakes in implementation, in other words. It’s much rarer to find a problem in the cryptographic algorithms (although that happens occasionally, like MD5). If we look through OpenSSL’s already-fixed vulnerabilities list:
https://www.openssl.org/news/vulnerabilities.html
Browser page search says that “buffer over” (as in buffer overflow, overrun, or over-read) appears on the page 24 times (although there’s double-counting there, as it tends to appear in the title and once or twice in the description). You don’t need to be a world-class security researcher to find these security holes; it’s more a matter of (a) happening to look in the right place, (b) having some knowledge and creativity and intelligence in figuring out how it could be exploited.
The NSA isn’t able to run thousands of copies of itself on machines it hacks into, nor use that to quickly create more powerful instances of itself. So that part of world domination is clearly out.
But if you mean “the NSA wouldn’t be able to do that first step of taking over millions of computers” (which the hypothetical AI would then use to bootstrap)… I disagree. Do you know about Stuxnet? Believed to be written by some combination of the NSA and Mossad. Observe:
So the NSA and/or Mossad had all these exploits and were sitting on them. Makes it plausible that, today, they have more that they’re sitting on.
The Conficker worm itself apparently did take over an estimated “9 million to 15 million” machines. So clearly that is doable.