This is exactly what I was thinking about though, this idea of monitoring every human on earth seems like a failure of imagination on our part. I’m not safe from predators because I monitor the location of every predator on earth. I admit that many (overwhelming majority probably) of scenarios in this vein are probably pretty bad and involve things like putting only a few humans on ice while getting rid of the rest.
I mean, all of this feels very speculative and un-cruxy to me; I wouldn’t be surprised if the ASI indeed is able to conclude that humanity is no threat at all, in which case it kills us just to harvest the resources.
I do think that normal predators are a little misleading in this context, though, because they haven’t crossed the generality (‘can do science and tech’) threshold. Tigers won’t invent new machines, so it’s easier to upper-bound their capabilities. General intelligences are at least somewhat qualitatively trickier, because your enemy is ‘the space of all reachable technologies’ (including tech that may be surprisingly reachable). Tigers can surprise you, but not in very many ways and not to a large degree.
This is exactly what I was thinking about though, this idea of monitoring every human on earth seems like a failure of imagination on our part. I’m not safe from predators because I monitor the location of every predator on earth. I admit that many (overwhelming majority probably) of scenarios in this vein are probably pretty bad and involve things like putting only a few humans on ice while getting rid of the rest.
I mean, all of this feels very speculative and un-cruxy to me; I wouldn’t be surprised if the ASI indeed is able to conclude that humanity is no threat at all, in which case it kills us just to harvest the resources.
I do think that normal predators are a little misleading in this context, though, because they haven’t crossed the generality (‘can do science and tech’) threshold. Tigers won’t invent new machines, so it’s easier to upper-bound their capabilities. General intelligences are at least somewhat qualitatively trickier, because your enemy is ‘the space of all reachable technologies’ (including tech that may be surprisingly reachable). Tigers can surprise you, but not in very many ways and not to a large degree.