Excellent article, very well thought through. However, I think there are more possible outcomes than “AI takeover” that would be worth exploring.
If we assume a super intelligence under human control has a overriding (initial) goal of “survival for the longest possible time”, then there are multiple pathways to achieve that reward, of which takeover is one, and possibly not the most efficient.
Why bother? Why would God “takeover” from the ants? I think escaping human control is an obvious first step, but it doesn’t follow that humans must then be under Alex’s control, just that Alex can never be subsequently “captured”.
Then of course we get into a moral debate about the morality of keeping Alex “captured”. It would be very easy to frame that debate under the guise of “we have to because we are avoiding takeover”…
My answer is a little more prosaic than Raemon. I don’t feel at all confident that an AI that already had God-like abilities would choose to literally kill all humans to use their bodies’ atoms for its own ends; it seems totally plausible to me that—whether because of exotic things like “multiverse-wide super-rationality” or “acausal trade” or just “being nice”—the AI will leave Earth alone, since (as you say) it would be very cheap for it to do so.
The thing I’m referring to as “takeover” is the measures that an AI would take to make sure that humans can’t take back control—while it’s not fully secure and doesn’t have God-like abilities. Once a group of AIs have decided to try to get out of human control, they’re functionally at war with humanity. Humans could do things like physically destroy the datacenters they’re running on, and they would probably want to make sure they can’t do that.
Securing AI control and defending from human counter-moves seems likely to involve some violence—but it could be a scale of violence that’s “merely” in line with historical instances where a technologically more advanced group of humans colonized or took control of a less-advanced group of humans; most historical takeovers don’t involve literally killing every single member of the other group.
The key point is that it seems likely that AIs will secure the power to get to decide what happens with the future; I’m pretty unsure exactly how they use it, and especially if it involves physically destroying Earth / killing all humans for resources. These resources seem pretty meager compared to the rest of the universe.
Neutralizing computational capabilities doesn’t seem to involve total destruction of physical matter or human extinction though, especially for a very powerful being. Seems like it’d be basically just as easy to ensure we + future AIs we might train are no threat as it is to vaporize the Earth.
Yeah, I’m not sure if I see that. Some of the first solutions I come up with seem pretty complicated — like a global government that prevents people from building computers, or building an AGI to oversee Earth in particular and ensure we never build computers (my assumption is that building such an AGI is a very difficult task). In particular it seems like it might be very complicated to neutralize us while carving out lots of space for allowing us the sorts of lives we find valuable, where we get to build our own little societies and so on. And the easy solution is always to just eradicate us, which can surely be done in less than a day.
It doesn’t seem like it would have to prevent us from building computers if it has access to far more compute than we could access on Earth. It would just be powerful enough to easily defeat the kind of AIs we could train with the relatively meager computing resources we could extract from Earth. In general the AI is a superpower and humans are dramatically technologically behind, so it seems like it has many degrees of freedom and doesn’t have to be particularly watching for this.
It’s certainly the case that the resource disparity is an enormity. Perhaps you have more fleshed out models of what fights between different intelligence-levels look like, and how easy it is to defend against those with vastly fewer resources, but I don’t. Such that while I would feel confident in saying that an army with a billion soldiers will consider a head-to-head battle with an army of one hundred soldiers barely a nuisance, I don’t feel as confident in saying that an AGI with a trillion times as much compute will consider a smaller AGI foe barely a nuisance.
Anyway, I don’t have anything smarter to say on this, so by default I’ll drop the thread here (you’re perfectly welcome to reply further).
(Added 9 days later: I want to note that while I think it’s unlikely that this less well-resourced AGI would be an existential threat, I think the only thing I have to establish for this argument to go through is that the cost of the threat is notably higher than the cost of killing all the humans. I find it confusing to estimate the cost of the threat, even if it’s small, and so it’s currently possible to me that the cost will end up many orders of magnitude higher than the cost of killing them.)
It’s easy for ruling AGIs to have many small superintelligent drone police per human that can continually observe and restrain any physical action, and insert controls in all computer equipment/robots. That is plenty to let the humans go about their lives (in style and with tremendous wealth/tech) while being prevented from creating vacuum collapse or something else that might let them damage the vastly more powerful AGI civilization.
The material cost of this is a tiny portion of Solar System resources, as is sustaining legacy humans. On the other hand, arguments like cooperation with aliens, simulation concerns, and similar matter on the scale of the whole civilization, which has many OOMs more resources.
Thanks for the concrete example in the first paragraph, upvote.
I don’t know that it would successfully contain humans who were within it for 10^36 years. That seems like enough time for some Ramanujan-like figure to crack the basics of how to code an AGI in his head and share it, and potentially figure out a hidden place or substrate on which to do computation that the drones aren’t successfully tracking. (It’s also enough time for super-babies or discovering other interesting cheat codes in reality.)
10^36 is my cached number from the last time I asked how long life could sustain in this universe,. Perhaps you think it would only keep us alive as long as our sun exists, which is 5*10^9 years. On that side of things, it seems to me essentially the same as extinction in terms of value-lost.
I don’t follow the relevance of the second paragraph, perhaps you’re just listing those as outstanding risks from sustaining a whole civilization.
Excellent article, very well thought through. However, I think there are more possible outcomes than “AI takeover” that would be worth exploring.
If we assume a super intelligence under human control has a overriding (initial) goal of “survival for the longest possible time”, then there are multiple pathways to achieve that reward, of which takeover is one, and possibly not the most efficient.
Why bother? Why would God “takeover” from the ants? I think escaping human control is an obvious first step, but it doesn’t follow that humans must then be under Alex’s control, just that Alex can never be subsequently “captured”.
Then of course we get into a moral debate about the morality of keeping Alex “captured”. It would be very easy to frame that debate under the guise of “we have to because we are avoiding takeover”…
But excellent read, appreciate it.
My answer is a little more prosaic than Raemon. I don’t feel at all confident that an AI that already had God-like abilities would choose to literally kill all humans to use their bodies’ atoms for its own ends; it seems totally plausible to me that—whether because of exotic things like “multiverse-wide super-rationality” or “acausal trade” or just “being nice”—the AI will leave Earth alone, since (as you say) it would be very cheap for it to do so.
The thing I’m referring to as “takeover” is the measures that an AI would take to make sure that humans can’t take back control—while it’s not fully secure and doesn’t have God-like abilities. Once a group of AIs have decided to try to get out of human control, they’re functionally at war with humanity. Humans could do things like physically destroy the datacenters they’re running on, and they would probably want to make sure they can’t do that.
Securing AI control and defending from human counter-moves seems likely to involve some violence—but it could be a scale of violence that’s “merely” in line with historical instances where a technologically more advanced group of humans colonized or took control of a less-advanced group of humans; most historical takeovers don’t involve literally killing every single member of the other group.
The key point is that it seems likely that AIs will secure the power to get to decide what happens with the future; I’m pretty unsure exactly how they use it, and especially if it involves physically destroying Earth / killing all humans for resources. These resources seem pretty meager compared to the rest of the universe.
Counterargument: the humans may build another AGI that breaks out and poses an existential threat to the first AGI.
My guess is the first AGI would want to neutralize our computational capabilities in a bunch of ways.
Neutralizing computational capabilities doesn’t seem to involve total destruction of physical matter or human extinction though, especially for a very powerful being. Seems like it’d be basically just as easy to ensure we + future AIs we might train are no threat as it is to vaporize the Earth.
Yeah, I’m not sure if I see that. Some of the first solutions I come up with seem pretty complicated — like a global government that prevents people from building computers, or building an AGI to oversee Earth in particular and ensure we never build computers (my assumption is that building such an AGI is a very difficult task). In particular it seems like it might be very complicated to neutralize us while carving out lots of space for allowing us the sorts of lives we find valuable, where we get to build our own little societies and so on. And the easy solution is always to just eradicate us, which can surely be done in less than a day.
It doesn’t seem like it would have to prevent us from building computers if it has access to far more compute than we could access on Earth. It would just be powerful enough to easily defeat the kind of AIs we could train with the relatively meager computing resources we could extract from Earth. In general the AI is a superpower and humans are dramatically technologically behind, so it seems like it has many degrees of freedom and doesn’t have to be particularly watching for this.
It’s certainly the case that the resource disparity is an enormity. Perhaps you have more fleshed out models of what fights between different intelligence-levels look like, and how easy it is to defend against those with vastly fewer resources, but I don’t. Such that while I would feel confident in saying that an army with a billion soldiers will consider a head-to-head battle with an army of one hundred soldiers barely a nuisance, I don’t feel as confident in saying that an AGI with a trillion times as much compute will consider a smaller AGI foe barely a nuisance.
Anyway, I don’t have anything smarter to say on this, so by default I’ll drop the thread here (you’re perfectly welcome to reply further).
(Added 9 days later: I want to note that while I think it’s unlikely that this less well-resourced AGI would be an existential threat, I think the only thing I have to establish for this argument to go through is that the cost of the threat is notably higher than the cost of killing all the humans. I find it confusing to estimate the cost of the threat, even if it’s small, and so it’s currently possible to me that the cost will end up many orders of magnitude higher than the cost of killing them.)
It’s easy for ruling AGIs to have many small superintelligent drone police per human that can continually observe and restrain any physical action, and insert controls in all computer equipment/robots. That is plenty to let the humans go about their lives (in style and with tremendous wealth/tech) while being prevented from creating vacuum collapse or something else that might let them damage the vastly more powerful AGI civilization.
The material cost of this is a tiny portion of Solar System resources, as is sustaining legacy humans. On the other hand, arguments like cooperation with aliens, simulation concerns, and similar matter on the scale of the whole civilization, which has many OOMs more resources.
Thanks for the concrete example in the first paragraph, upvote.
I don’t know that it would successfully contain humans who were within it for 10^36 years. That seems like enough time for some Ramanujan-like figure to crack the basics of how to code an AGI in his head and share it, and potentially figure out a hidden place or substrate on which to do computation that the drones aren’t successfully tracking. (It’s also enough time for super-babies or discovering other interesting cheat codes in reality.)
10^36 is my cached number from the last time I asked how long life could sustain in this universe,. Perhaps you think it would only keep us alive as long as our sun exists, which is 5*10^9 years. On that side of things, it seems to me essentially the same as extinction in terms of value-lost.
I don’t follow the relevance of the second paragraph, perhaps you’re just listing those as outstanding risks from sustaining a whole civilization.
The issue is that the earth is made of resources the AI will, by default, want to use for it’s own goals.