Neutralizing computational capabilities doesn’t seem to involve total destruction of physical matter or human extinction though, especially for a very powerful being. Seems like it’d be basically just as easy to ensure we + future AIs we might train are no threat as it is to vaporize the Earth.
Yeah, I’m not sure if I see that. Some of the first solutions I come up with seem pretty complicated — like a global government that prevents people from building computers, or building an AGI to oversee Earth in particular and ensure we never build computers (my assumption is that building such an AGI is a very difficult task). In particular it seems like it might be very complicated to neutralize us while carving out lots of space for allowing us the sorts of lives we find valuable, where we get to build our own little societies and so on. And the easy solution is always to just eradicate us, which can surely be done in less than a day.
It doesn’t seem like it would have to prevent us from building computers if it has access to far more compute than we could access on Earth. It would just be powerful enough to easily defeat the kind of AIs we could train with the relatively meager computing resources we could extract from Earth. In general the AI is a superpower and humans are dramatically technologically behind, so it seems like it has many degrees of freedom and doesn’t have to be particularly watching for this.
It’s certainly the case that the resource disparity is an enormity. Perhaps you have more fleshed out models of what fights between different intelligence-levels look like, and how easy it is to defend against those with vastly fewer resources, but I don’t. Such that while I would feel confident in saying that an army with a billion soldiers will consider a head-to-head battle with an army of one hundred soldiers barely a nuisance, I don’t feel as confident in saying that an AGI with a trillion times as much compute will consider a smaller AGI foe barely a nuisance.
Anyway, I don’t have anything smarter to say on this, so by default I’ll drop the thread here (you’re perfectly welcome to reply further).
(Added 9 days later: I want to note that while I think it’s unlikely that this less well-resourced AGI would be an existential threat, I think the only thing I have to establish for this argument to go through is that the cost of the threat is notably higher than the cost of killing all the humans. I find it confusing to estimate the cost of the threat, even if it’s small, and so it’s currently possible to me that the cost will end up many orders of magnitude higher than the cost of killing them.)
It’s easy for ruling AGIs to have many small superintelligent drone police per human that can continually observe and restrain any physical action, and insert controls in all computer equipment/robots. That is plenty to let the humans go about their lives (in style and with tremendous wealth/tech) while being prevented from creating vacuum collapse or something else that might let them damage the vastly more powerful AGI civilization.
The material cost of this is a tiny portion of Solar System resources, as is sustaining legacy humans. On the other hand, arguments like cooperation with aliens, simulation concerns, and similar matter on the scale of the whole civilization, which has many OOMs more resources.
Thanks for the concrete example in the first paragraph, upvote.
I don’t know that it would successfully contain humans who were within it for 10^36 years. That seems like enough time for some Ramanujan-like figure to crack the basics of how to code an AGI in his head and share it, and potentially figure out a hidden place or substrate on which to do computation that the drones aren’t successfully tracking. (It’s also enough time for super-babies or discovering other interesting cheat codes in reality.)
10^36 is my cached number from the last time I asked how long life could sustain in this universe,. Perhaps you think it would only keep us alive as long as our sun exists, which is 5*10^9 years. On that side of things, it seems to me essentially the same as extinction in terms of value-lost.
I don’t follow the relevance of the second paragraph, perhaps you’re just listing those as outstanding risks from sustaining a whole civilization.
Counterargument: the humans may build another AGI that breaks out and poses an existential threat to the first AGI.
My guess is the first AGI would want to neutralize our computational capabilities in a bunch of ways.
Neutralizing computational capabilities doesn’t seem to involve total destruction of physical matter or human extinction though, especially for a very powerful being. Seems like it’d be basically just as easy to ensure we + future AIs we might train are no threat as it is to vaporize the Earth.
Yeah, I’m not sure if I see that. Some of the first solutions I come up with seem pretty complicated — like a global government that prevents people from building computers, or building an AGI to oversee Earth in particular and ensure we never build computers (my assumption is that building such an AGI is a very difficult task). In particular it seems like it might be very complicated to neutralize us while carving out lots of space for allowing us the sorts of lives we find valuable, where we get to build our own little societies and so on. And the easy solution is always to just eradicate us, which can surely be done in less than a day.
It doesn’t seem like it would have to prevent us from building computers if it has access to far more compute than we could access on Earth. It would just be powerful enough to easily defeat the kind of AIs we could train with the relatively meager computing resources we could extract from Earth. In general the AI is a superpower and humans are dramatically technologically behind, so it seems like it has many degrees of freedom and doesn’t have to be particularly watching for this.
It’s certainly the case that the resource disparity is an enormity. Perhaps you have more fleshed out models of what fights between different intelligence-levels look like, and how easy it is to defend against those with vastly fewer resources, but I don’t. Such that while I would feel confident in saying that an army with a billion soldiers will consider a head-to-head battle with an army of one hundred soldiers barely a nuisance, I don’t feel as confident in saying that an AGI with a trillion times as much compute will consider a smaller AGI foe barely a nuisance.
Anyway, I don’t have anything smarter to say on this, so by default I’ll drop the thread here (you’re perfectly welcome to reply further).
(Added 9 days later: I want to note that while I think it’s unlikely that this less well-resourced AGI would be an existential threat, I think the only thing I have to establish for this argument to go through is that the cost of the threat is notably higher than the cost of killing all the humans. I find it confusing to estimate the cost of the threat, even if it’s small, and so it’s currently possible to me that the cost will end up many orders of magnitude higher than the cost of killing them.)
It’s easy for ruling AGIs to have many small superintelligent drone police per human that can continually observe and restrain any physical action, and insert controls in all computer equipment/robots. That is plenty to let the humans go about their lives (in style and with tremendous wealth/tech) while being prevented from creating vacuum collapse or something else that might let them damage the vastly more powerful AGI civilization.
The material cost of this is a tiny portion of Solar System resources, as is sustaining legacy humans. On the other hand, arguments like cooperation with aliens, simulation concerns, and similar matter on the scale of the whole civilization, which has many OOMs more resources.
Thanks for the concrete example in the first paragraph, upvote.
I don’t know that it would successfully contain humans who were within it for 10^36 years. That seems like enough time for some Ramanujan-like figure to crack the basics of how to code an AGI in his head and share it, and potentially figure out a hidden place or substrate on which to do computation that the drones aren’t successfully tracking. (It’s also enough time for super-babies or discovering other interesting cheat codes in reality.)
10^36 is my cached number from the last time I asked how long life could sustain in this universe,. Perhaps you think it would only keep us alive as long as our sun exists, which is 5*10^9 years. On that side of things, it seems to me essentially the same as extinction in terms of value-lost.
I don’t follow the relevance of the second paragraph, perhaps you’re just listing those as outstanding risks from sustaining a whole civilization.