New-to-me thought I had in response to the kill all humans part. When predators are a threat to you, you of course shoot them. But once you invent cheap tech that can control them you don’t need to kill them anymore. The story goes that the AI would kill us either because we are a threat or because we are irrelevant. It seems to me that (and this imports a bunch of extra stuff that would require analysis to turn this into a serious analysis, this is just an idle thought), the first thing I do if I am superintelligent and wanting to secure my position is not take over the earth, which isn’t in a particularly useful spot resource wise and instead launch my nanofactory beyond the reach of humans to mercury or something. Similarly, in the nanomachines in everyone’s blood that can kill them instantly class of ideas, why do I need at that point to actually pull the switch? I.e. the kill all humans scenario is emotionally salient but doesn’t actually clearly follow the power gradients that you want to climb for instrumental convergence reasons?
If humans were able to make one super-powerful AI, then humans would probably be able to make a second super-powerful AI, with different goals, which would then compete with the first AI. Unless, of course, the humans are somehow prevented from making more AIs, e.g. because they’re all dead.
I guess the threat model relies on the overhang. If you need x compute for powerful ai, then you need to control more than all the compute on earth minus x to ensure safety, or something like that. Controlling the people probably much easier.
But once you invent cheap tech that can control them you don’t need to kill them anymore.
A paperclipper mainly cares about humans because we might have some way to threaten the paperclipper (e.g., by pushing a button that deploys a rival superintelligence); and secondarily, we’re made of atoms that can be used to build paperclips.
It’s harder to monitor the actions of every single human on Earth, than it is to kill all humans; and there’s a risk that monitoring people visibly will cause someone to push the ‘deploy a rival superintelligence’ button, if such a button exists.
Also, every minute that passes without you killing all humans, in the time window between ‘I’m confident I can kill all humans’ and ‘I’m carefully surveilling every human on Earth and know that there’s no secret bunker where someone has a Deploy Superintelligence button’, is a minute where you’re risking somebody pushing the ‘deploy a rival superintelligence’ button. This makes me think that the value of delaying ‘killing all humans’ (once you’re confident you can do it) would need to be very high in order to offset that risk.
One reason I might be wrong is if the AGI is worried about something like a dead man’s switch that deploys a rival superintelligence iff some human isn’t alive and regularly performing some action. (Not necessarily a likely scenario on priors, but once you’re confident enough in your base plan, unlikely scenarios can end up dominating the remaining scenarios where you lose.) Then it’s at least possible that you’d want to delay long enough to confirm that no such switch exists.
the first thing I do if I am superintelligent and wanting to secure my position is not take over the earth, which isn’t in a particularly useful spot resource wise and instead launch my nanofactory beyond the reach of humans to mercury or something.
You should be able to do both in parallel. I don’t have a strong view on which is higher-priority. Given the dead-man’s-switch worry above, you might want to prioritize sending a probe off-planet first as a precaution; but then go ahead and kill humans ASAP.
This is exactly what I was thinking about though, this idea of monitoring every human on earth seems like a failure of imagination on our part. I’m not safe from predators because I monitor the location of every predator on earth. I admit that many (overwhelming majority probably) of scenarios in this vein are probably pretty bad and involve things like putting only a few humans on ice while getting rid of the rest.
I mean, all of this feels very speculative and un-cruxy to me; I wouldn’t be surprised if the ASI indeed is able to conclude that humanity is no threat at all, in which case it kills us just to harvest the resources.
I do think that normal predators are a little misleading in this context, though, because they haven’t crossed the generality (‘can do science and tech’) threshold. Tigers won’t invent new machines, so it’s easier to upper-bound their capabilities. General intelligences are at least somewhat qualitatively trickier, because your enemy is ‘the space of all reachable technologies’ (including tech that may be surprisingly reachable). Tigers can surprise you, but not in very many ways and not to a large degree.
You don’t need to kill them, but it’s still helpful. There could be a moment where it’s a better investment to send stuff into some temporarily unreachable spot like Mercury or the bottom of the ocean, than to kill everything, though I don’t see practically how you could send something to the bottom of the ocean that would carry on your goals (a nanofactory programmed to make computers and run a copy of your source code, say) without also being able to easily kill everything on Earth. But regardless, soon after that moment, you’re able to kill everything, and that’s still a CIG.
I suspect a sufficiently intelligent, unaligned artificial intelligence would both kill us all immediately, and immediately start expanding its reach in all directions of space at near light speed. There is no reason for there to be an either-or.
Knowing you came from neuromorphic architecture, and other than humans being threatening to you, why would you destroy the most complex thing you are aware of? Sure, maybe you put a few humans on ice and get rid of the rest.
I agree it’s plausible that a paperclip maximizer would destructively scan a human or two and keep the scan around for some length of time. Though I’d guess this has almost no effect on the future’s long-term EV.
New-to-me thought I had in response to the kill all humans part. When predators are a threat to you, you of course shoot them. But once you invent cheap tech that can control them you don’t need to kill them anymore. The story goes that the AI would kill us either because we are a threat or because we are irrelevant. It seems to me that (and this imports a bunch of extra stuff that would require analysis to turn this into a serious analysis, this is just an idle thought), the first thing I do if I am superintelligent and wanting to secure my position is not take over the earth, which isn’t in a particularly useful spot resource wise and instead launch my nanofactory beyond the reach of humans to mercury or something. Similarly, in the nanomachines in everyone’s blood that can kill them instantly class of ideas, why do I need at that point to actually pull the switch? I.e. the kill all humans scenario is emotionally salient but doesn’t actually clearly follow the power gradients that you want to climb for instrumental convergence reasons?
If humans were able to make one super-powerful AI, then humans would probably be able to make a second super-powerful AI, with different goals, which would then compete with the first AI. Unless, of course, the humans are somehow prevented from making more AIs, e.g. because they’re all dead.
I guess the threat model relies on the overhang. If you need x compute for powerful ai, then you need to control more than all the compute on earth minus x to ensure safety, or something like that. Controlling the people probably much easier.
Yes, where killing all humans is an example of “controlling the people”, from the perspective of an Unfriendly AI.
A paperclipper mainly cares about humans because we might have some way to threaten the paperclipper (e.g., by pushing a button that deploys a rival superintelligence); and secondarily, we’re made of atoms that can be used to build paperclips.
It’s harder to monitor the actions of every single human on Earth, than it is to kill all humans; and there’s a risk that monitoring people visibly will cause someone to push the ‘deploy a rival superintelligence’ button, if such a button exists.
Also, every minute that passes without you killing all humans, in the time window between ‘I’m confident I can kill all humans’ and ‘I’m carefully surveilling every human on Earth and know that there’s no secret bunker where someone has a Deploy Superintelligence button’, is a minute where you’re risking somebody pushing the ‘deploy a rival superintelligence’ button. This makes me think that the value of delaying ‘killing all humans’ (once you’re confident you can do it) would need to be very high in order to offset that risk.
One reason I might be wrong is if the AGI is worried about something like a dead man’s switch that deploys a rival superintelligence iff some human isn’t alive and regularly performing some action. (Not necessarily a likely scenario on priors, but once you’re confident enough in your base plan, unlikely scenarios can end up dominating the remaining scenarios where you lose.) Then it’s at least possible that you’d want to delay long enough to confirm that no such switch exists.
You should be able to do both in parallel. I don’t have a strong view on which is higher-priority. Given the dead-man’s-switch worry above, you might want to prioritize sending a probe off-planet first as a precaution; but then go ahead and kill humans ASAP.
This is exactly what I was thinking about though, this idea of monitoring every human on earth seems like a failure of imagination on our part. I’m not safe from predators because I monitor the location of every predator on earth. I admit that many (overwhelming majority probably) of scenarios in this vein are probably pretty bad and involve things like putting only a few humans on ice while getting rid of the rest.
I mean, all of this feels very speculative and un-cruxy to me; I wouldn’t be surprised if the ASI indeed is able to conclude that humanity is no threat at all, in which case it kills us just to harvest the resources.
I do think that normal predators are a little misleading in this context, though, because they haven’t crossed the generality (‘can do science and tech’) threshold. Tigers won’t invent new machines, so it’s easier to upper-bound their capabilities. General intelligences are at least somewhat qualitatively trickier, because your enemy is ‘the space of all reachable technologies’ (including tech that may be surprisingly reachable). Tigers can surprise you, but not in very many ways and not to a large degree.
You don’t need to kill them, but it’s still helpful. There could be a moment where it’s a better investment to send stuff into some temporarily unreachable spot like Mercury or the bottom of the ocean, than to kill everything, though I don’t see practically how you could send something to the bottom of the ocean that would carry on your goals (a nanofactory programmed to make computers and run a copy of your source code, say) without also being able to easily kill everything on Earth. But regardless, soon after that moment, you’re able to kill everything, and that’s still a CIG.
I suspect a sufficiently intelligent, unaligned artificial intelligence would both kill us all immediately, and immediately start expanding its reach in all directions of space at near light speed. There is no reason for there to be an either-or.
Knowing you came from neuromorphic architecture, and other than humans being threatening to you, why would you destroy the most complex thing you are aware of? Sure, maybe you put a few humans on ice and get rid of the rest.
I agree it’s plausible that a paperclip maximizer would destructively scan a human or two and keep the scan around for some length of time. Though I’d guess this has almost no effect on the future’s long-term EV.