The nanobot thing is not a crux whatsoever. If you have enough cognitive power, you have a gazillion avenues to destroy an intellectually inferior and oblivious foe.
Take just the domain of computer security. Our computer networks and software are piles of abstractions built atop one another. Nowadays we humans barely understand them, and certainly can’t secure them, which is why cyber crime works. Human hackers can e.g. steal large amounts of cryptocurrency; an entity with more cognitive power could more easily steal larger amounts. Or do large-scale ransomware attacks. Or take over bot farms to increase its computing power. And so on. Now it has cognitive power and tons of resources in the form of computing power and money, for whatever steps it wants to take next.
It still needs access to weapons it can use to wipe out humanity. It could try to pay people to build dangerous things for it, or convince its owners to pay for them, of course. What are you imagining it doing? Nukes? Slaughterbots? Bio/chemical agents? Which ones is it very likely to get past security to access or build without raising alarms and being prevented? And say it gets such weapons. How does it deliver them to wipe out humanity, given our defenses?
It also doesn’t yet have the physical power to keep itself from being shut down on those computers it hacked in your scenario. I think large illicit computations on powerful computers are reasonably likely to be noticed, and distributing computations into small chunks to run across a huge number of, say personal computers/laptops, will plausibly be very slow, due to frequent transfer over the internet.
However, it could plausibly just pay for cloud computing without raising alarms if it builds wealth first.
What current defenses do you think we have against nukes or pandemics?
For instance, the lesson from Covid seems to be that a small group of humans is already enough to trigger a pandemic. If one intended to develop an especially lethal pandemic via gain-of-function research, the task already doesn’t seem particularly hard for researchers with time and resources, so we’d expect a superintelligence to have a much easier job.
If getting access to nukes via hacking seems too implausible, then maybe it’s easier to imagine triggering nuclear war by tricking one nuclear power into thinking it’s under attack by another. We’ve had close calls in the past merely due to bad sensors!
More generally, given all the various x-risks we already think about, I just don’t consider humanity in its current position to be particularly secure. And that’s our current position, minus an adversary who could optimize the situation towards our extinction.
Regarding the safety of the AGI, you’d expect it not to do things that get it noticed until it’s sufficiently safe. So you’d expect it to only get noticed if it believes it can get away with it. I also think our civilization clearly lacks the ability to coordinate to e.g. turn off the Internet or something, if that was necessary to stop an AGI once it had reached the point of distributed computation.
Personal protective equipment and isolation can protect against infectious disease, at the very least. A more deadly and infectious virus than COVID would be taken far more seriously.
I think nuclear war is unlikely to wipe out humanity, since there are enough countries that are unlikely targets, and I don’t think all of the US would be wiped out anyway. I’m less sure about nuclear winter, but those in the community who’ve done research on it seem skeptical that it would wipe us out. Maybe it reduces the population enough for an AGI to target the rest of us or prevent us from rebuilding, though.
Some posts here:
https://forum.effectivealtruism.org/topics/nuclear-warfare-1https://forum.effectivealtruism.org/topics/nuclear-winter
Maybe it reduces the population enough for an AGI to target the rest of us or prevent us from rebuilding, though.
Yeah, I’m familiar with the arguments that neither pandemics nor nuclear war seem likely to be existential risks, i.e. ones that could cause human extinction; but I’d nonetheless expect such events to be damaging enough from the perspective of a nefarious actor trying to prevent resistance.
Ultimately this whole line of reasoning seems superfluous to me—it just seems so obvious that with sufficient cognitive power one can do ridiculous things—but for those who trip up on the suggested nanotech stuff, maybe a more palatable argument is: You know those other x-risks you’re already worrying about? A sufficiently intelligent antagonist can exacerbate those nigh-arbitrarily.
To be clear: I am not saying that an AGI won’t be dangerous, that an AGI won’t be much clever than us or that it is not worth working on AGI safety. I am saying that I believe that an AGI could not theoretically kill all humans because it is not only a matter of being very intelligent.
You claim that superintelligence is not enough to wipe out humanity, and I’m saying that superintelligence trivially gets you resources. If you think that superintelligence and resources are still not enough to wipe out humanity, what more do you want?
What about plans like “hack cryptocurrency for coins worth hundreds of millions of dollars” or “make ransomware attacks” is not trivial? Cybercrimes like these are regularly committed by humans, and so a superintelligence will naturally have a much easier time with them.
If we postulate a superintelligence with nothing but Internet access, it should be many orders of magnitude better at making money in the pure Internet economy (e.g. cybercrime, cryptocurrency, lots of investment stuff, online gambling, prediction markets) than humans are, and some humans already make a lot of money there.
Oh yes, I don’t have any issues with a plan where the machine hacks crypto, though I am not sure how capable would be of doing that without raising any alarms from any group in the world, how it could guarantee that someone is not monitoring it. After that, remember you still need a lot of inferential steps to get to a point where you successfully deploy those cryptos into things that can exterminate humans. And keep in mind that you need to do that without being discovered and in a super short amount of time.
And keep in mind that you need to do that without being discovered and in a super short amount of time.
While I expect that this would be the case, I don’t consider it a crux. As long as the AGI can keep itself safe, it doesn’t particularly matter if it’s discovered, as long as it has become powerful enough, and/or distributed enough, that our civilization can no longer stop it. And given our civilization’s level of competence, those are low bars to clear.
The nanobot thing is not a crux whatsoever. If you have enough cognitive power, you have a gazillion avenues to destroy an intellectually inferior and oblivious foe.
Take just the domain of computer security. Our computer networks and software are piles of abstractions built atop one another. Nowadays we humans barely understand them, and certainly can’t secure them, which is why cyber crime works. Human hackers can e.g. steal large amounts of cryptocurrency; an entity with more cognitive power could more easily steal larger amounts. Or do large-scale ransomware attacks. Or take over bot farms to increase its computing power. And so on. Now it has cognitive power and tons of resources in the form of computing power and money, for whatever steps it wants to take next.
It still needs access to weapons it can use to wipe out humanity. It could try to pay people to build dangerous things for it, or convince its owners to pay for them, of course. What are you imagining it doing? Nukes? Slaughterbots? Bio/chemical agents? Which ones is it very likely to get past security to access or build without raising alarms and being prevented? And say it gets such weapons. How does it deliver them to wipe out humanity, given our defenses?
It also doesn’t yet have the physical power to keep itself from being shut down on those computers it hacked in your scenario. I think large illicit computations on powerful computers are reasonably likely to be noticed, and distributing computations into small chunks to run across a huge number of, say personal computers/laptops, will plausibly be very slow, due to frequent transfer over the internet.
However, it could plausibly just pay for cloud computing without raising alarms if it builds wealth first.
What current defenses do you think we have against nukes or pandemics?
For instance, the lesson from Covid seems to be that a small group of humans is already enough to trigger a pandemic. If one intended to develop an especially lethal pandemic via gain-of-function research, the task already doesn’t seem particularly hard for researchers with time and resources, so we’d expect a superintelligence to have a much easier job.
If getting access to nukes via hacking seems too implausible, then maybe it’s easier to imagine triggering nuclear war by tricking one nuclear power into thinking it’s under attack by another. We’ve had close calls in the past merely due to bad sensors!
More generally, given all the various x-risks we already think about, I just don’t consider humanity in its current position to be particularly secure. And that’s our current position, minus an adversary who could optimize the situation towards our extinction.
Regarding the safety of the AGI, you’d expect it not to do things that get it noticed until it’s sufficiently safe. So you’d expect it to only get noticed if it believes it can get away with it. I also think our civilization clearly lacks the ability to coordinate to e.g. turn off the Internet or something, if that was necessary to stop an AGI once it had reached the point of distributed computation.
Personal protective equipment and isolation can protect against infectious disease, at the very least. A more deadly and infectious virus than COVID would be taken far more seriously.
I think nuclear war is unlikely to wipe out humanity, since there are enough countries that are unlikely targets, and I don’t think all of the US would be wiped out anyway. I’m less sure about nuclear winter, but those in the community who’ve done research on it seem skeptical that it would wipe us out. Maybe it reduces the population enough for an AGI to target the rest of us or prevent us from rebuilding, though. Some posts here: https://forum.effectivealtruism.org/topics/nuclear-warfare-1 https://forum.effectivealtruism.org/topics/nuclear-winter
Yeah, I’m familiar with the arguments that neither pandemics nor nuclear war seem likely to be existential risks, i.e. ones that could cause human extinction; but I’d nonetheless expect such events to be damaging enough from the perspective of a nefarious actor trying to prevent resistance.
Ultimately this whole line of reasoning seems superfluous to me—it just seems so obvious that with sufficient cognitive power one can do ridiculous things—but for those who trip up on the suggested nanotech stuff, maybe a more palatable argument is: You know those other x-risks you’re already worrying about? A sufficiently intelligent antagonist can exacerbate those nigh-arbitrarily.
To be clear: I am not saying that an AGI won’t be dangerous, that an AGI won’t be much clever than us or that it is not worth working on AGI safety. I am saying that I believe that an AGI could not theoretically kill all humans because it is not only a matter of being very intelligent.
Typo? (could not kill all humans)
Typo
The thing is, I don’t really disagree with this. Can you read again what I am arguing against?
You claim that superintelligence is not enough to wipe out humanity, and I’m saying that superintelligence trivially gets you resources. If you think that superintelligence and resources are still not enough to wipe out humanity, what more do you want?
Well, if you say that it trivially gets your resources, we do have a crux.
What about plans like “hack cryptocurrency for coins worth hundreds of millions of dollars” or “make ransomware attacks” is not trivial? Cybercrimes like these are regularly committed by humans, and so a superintelligence will naturally have a much easier time with them.
If we postulate a superintelligence with nothing but Internet access, it should be many orders of magnitude better at making money in the pure Internet economy (e.g. cybercrime, cryptocurrency, lots of investment stuff, online gambling, prediction markets) than humans are, and some humans already make a lot of money there.
Oh yes, I don’t have any issues with a plan where the machine hacks crypto, though I am not sure how capable would be of doing that without raising any alarms from any group in the world, how it could guarantee that someone is not monitoring it. After that, remember you still need a lot of inferential steps to get to a point where you successfully deploy those cryptos into things that can exterminate humans. And keep in mind that you need to do that without being discovered and in a super short amount of time.
While I expect that this would be the case, I don’t consider it a crux. As long as the AGI can keep itself safe, it doesn’t particularly matter if it’s discovered, as long as it has become powerful enough, and/or distributed enough, that our civilization can no longer stop it. And given our civilization’s level of competence, those are low bars to clear.