I think it’s plausible the A.I. would reshape the world but not in a way that would kill us, at least not for a long time—and not because it cares about us a little, or because of acausal incentives, or because it won’t be that powerful (though @paulfchristiano’s story about this is somewhat likely and adds to mine more or less disjunctively).
If this seems impossible to you, perhaps you’re imagining a gray goo scenario as the central outcome. But that is a very questionable assumption, and I think it is load bearing—if the A.G.I. does something more “clever” than absolutely crack nanotech, either because nanotech is hard OR because it sees an easier strategy, I think we don’t necessarily die.
We don’t trade with ants, but we also don’t exterminate all ants. Partially this is because we can’t productively burn their bodies for energy, but actually ants are annoying and many people would like to get rid of them. It’s just not worth the effort to get rid of them, it would be going out of our way.
Similarly, an A.G.I. would see different opportunities and constraints than we do. It is a fairly defensible assumption that it would still want to grab as much negentropy and energy as possible—this underlies the same model of computation in our universe that makes me think A.G.I. will happen in the first place, so I’m not highly inclined to question it. However, it is not necessarily the case that the most efficient route to getting more energy factors through using all energy that is available locally. Burning the biosphere in Antarctica might just require too much legwork to be optimal. Once the most easily extractable energy on Earth is extracted, an A.G.I. might invest it in e.g. taking apart Mercury to build a Dyson sphere. I guess it’s important to be clear about what I mean by “legwork” here—I expect an A.G.I. to only have so many “workers” / actuators / “arms” at a given time, and to only be able to pay attention to so many things. Perhaps it is better to focus on designing Dyson sphere technology than biosphere burning technology. I don’t see how legwork ceases to be a factor unless you believe in gray goo.
Early in A.G.I. takeover, I expect the infrastructure of humanity to be useful because an A.G.I. will not have designed radically superior infrastructure. It will probably focus on designing infrastructure for things we can’t do at all and continue to borrow our existing infrastructure for the things we are already doing because redesigning and going through the effort of replacing it isn’t the most valuable course of action on the margin.
Later in A.G.I. takeover, it will be mainly exploiting resources we don’t have access to. This probably means eating stars, but it’s possible (even likely) that our model of the incentives for Gods are not accurate. Jaynes expected less conflict between beings with radically different utility functions—obviously this is false when interpreted to mean “radically different goals for the universe” but I think it may hold up in the context he originally meant it, when different people want access to different resources. If Dyson sphere level takes awhile to achieve, and in the meantime A.G.I. has been trading with humans, human society will be much more productive, and A.G.I. will care much LESS about exploiting the remaining resources on Earth as efficiently as possible. So trade may continue at this point.
There are two obvious objections, and I think the first is persuasive and the second isn’t.
1: The A.G.I. will eventually take apart our sun which will kill us.
Continuing under the assumption that the A.G.I. cares about us literally not at all and has complete control (that is, conditioning on the story I’ve told above) I think this is very likely but still not certain. For one thing, I am only say 98% sure an A.G.I. would still care about getting more energy at this stage, and not something else we have no name for. For another, if trade really has continued with the increasingly competent society of humans on Earth, we might not need the sun to survive—we could get some batteries in exchange for doing very low priority homework sets etc., and we could use geothermal. Presumably the sun would still be eventually dissassembled, and we probably die not so long after that point (I guess we could try changing to a star the A.G.I. isn’t using), but it’s not obvious to me that even Gods eat stars in less than centuries. Maybe you get more energy out if you eat them slowly, maybe it’s just hard to eat them fast. So those of us currently alive don’t necessarily suffer untimely deaths.
2: The A.G.I. will kill us so we don’t build another A.G.I.
No need—if our society is useful it would be very easy to grab all the GPU’s from us without killing us. And the faster the takeoff, the less threat an A.G.I. created later (say, without GPU’s) would pose—whereas as slow takeoff is favorable for other reasons. I think there are actually a very narrow range of possibilities where A.G.I. kills us ONLY because it’s afraid we’ll build another A.G.I. We aren’t nearly competent enough to get away with that in real life.
(As another interesting but unrelated note on the fragility of value: though some humans may try to remove the suffering from nature, if I get a fraction of the lightcone some true wilderness will remain. So I don’t buy the example @So8res chose an I’m not sure I buy into the ultra-strong version of fragility of value.)
If AGI bulds Dyson sphere, we are dead from the simple fact of not having sunlight.
Technology of disassembling Mercury is not different from technology of disassembling Moon/Earth and easy to use to kill everyone—you just shoot relativistic projectiles using electromagnetic propulsion and evaporate swatches of planetary crust.
2: Yes, but once Dyson sphere building tech is available I am not sure dissassembling Earth will be useful on the margin. I think Mercury provides sufficient raw materials to build a Dyson sphere and far more energy can be extracted by optimizing the Dyson sphere or hopping to other stars than grabbing the tiny amount available on Earth. Also, Earth is already home to a lot of well developed infrastructure. To the extent that takeoff looks more Hansonian than Yudkowskian, this infrastructure will become much more valuable during takeoff, and ripping it up for parts may not be wise.
My intuition is that Earth would probably be destroyed, but I think it’s worth pointing out that the economic calculation isn’t actually trivial. It seems that most rationalists expect an A.G.I. to sort of omnipotently grab all resources in the lightcone, but perhaps it would still face tradeoffs and need to prioritize—and this includes potentially pursuing opportunities we aren’t even aware of, which may not interfere with us at all.
redesigning and going through the effort of replacing it isn’t the most valuable course of action on the margin.
Such effort would most likely be a trivial expenditure compared to the resources those actions are about acquiring, and wouldn’t be as likely to entail significant opportunity costs as in the case of humans taking those actions, as AIs could parallelize their efforts when needed.
The number of Von Neumann probes one can produce should go up the more planetary material is used, so I’m not sure the adequacy of Mercury helps much. If one produces fewer probes, the expansion time (while still an exponential) starts out much slower, and at any given time growth rate would be significantly lower than it otherwise would have been.
There is a large disjunction of possible optimal behaviors, and some of these might be pursued simultaneously for the sake of avoiding risks by reserving options. Most things that look like making optimal use of resources in our solar system without considering human values are going to kill all humans.
it’s not obvious to me that even Gods eat stars in less than centuries
Same, but it’d be about what portion of the sun’s output is captured, not rate of disassembly.
I expect an A.G.I. to only have so many “workers” / actuators / “arms” at a given time, and to only be able to pay attention to so many things
If this were a significant bottleneck, building new actuators or running in parallel to avoid attentional limitations would be made a high priority. I wouldn’t expect a capable AI to be significantly limited in this way for long.
I am only say 98% sure an A.G.I. would still care about getting more energy at this stage, and not something else we have no name for.
An AI might not want to be highly visible to the cosmic environment and so not dim the star noticeably, or stand to get much more from acausal trade (these would still usually entail using the local resources optimally relative to those trades), or have access to negentropy stores far more vast than entailed by exploiting large celestial bodies (but what could cause the system to become fully neutral to the previously accessible resources? It would be tremendously surprising to not entail using or dissipating those resources so no competitors can arise from their use.) More energy would most likely mean earlier starts on any critical phases of its plan(s), better ability to conclude plans will work, and better ability to verify plans have worked.
the economic calculation isn’t actually trivial
True, but some parts of the situation are easier to predict than others, e.g. there’s a conjunction of many factors necessary to support human life (surface temperature as influenced by the sun / compute / atmosphere, lack of disassembly for resources, atmospheric toxicity / presence at all, strength of Earth’s magnetic field, etc), and conditioned on extreme scale unaligned AI projects that would plausibly touch many of these critical factors, the probability of survival comes out quite low for most settings of how it could go about them.
if trade really has continued with the increasingly competent society of humans on Earth, we might not need the sun to survive
I think there are actually a very narrow range of possibilities where A.G.I. kills us ONLY because it’s afraid we’ll build another A.G.I. We aren’t nearly competent enough to get away with that in real life.
If we’re conditioning on getting an unaligned ASI, and humans are still trying to produce a friendly competitor, this seems highly likely to result in being squished. In that scenario, we’d already be conditioning on having been able to build a first AGI, so a second becomes highly probable.
The most plausible versions to me entail behaviors that either don’t look like they’re considering the presence of humans (because they don’t need to) and result in everyone dying, or are optimally exploiting the presence of humans via short-term persuasion and then repurposing humans for acausal trade scenarios or discarding them. It does seem fair to doubt we’d be given an opportunity to build a competitor, but humanity in a condition where it is unable to build AI for reasons other than foresight seems overwhelmingly likely to entail doom.
While we could be surprised by the outcome, and possibly for reasons you’ve mentioned, it still seems most probable that (given an unaligned capable AI) very capable grabbing of resources in ways that kill humans would occur, and that many rationalists are mostly working from the right model there.
I would be modestly surprised, but not very surprised, if an A.G.I. could cause build a Dyson sphere causing the sun to be dimmed by >20% in less than a couple decades (I think a few percent isn’t enough to cause crop failure), but within a century is plausible to me.
I don’t think we would be squashed for our potential to build a competitor. I think that a competitor would no longer be a serious threat once an A.G.I. seized all available compute.
I give a little more credence to various “unknown unknowns” about the laws of physics and the priorities of superintelligences implying that an A.G.I. would no longer care to exploit the resources we need.
Overall rationalists are right to worry about being killed by A.G.I.
I think it’s plausible the A.I. would reshape the world but not in a way that would kill us, at least not for a long time—and not because it cares about us a little, or because of acausal incentives, or because it won’t be that powerful (though @paulfchristiano’s story about this is somewhat likely and adds to mine more or less disjunctively).
If this seems impossible to you, perhaps you’re imagining a gray goo scenario as the central outcome. But that is a very questionable assumption, and I think it is load bearing—if the A.G.I. does something more “clever” than absolutely crack nanotech, either because nanotech is hard OR because it sees an easier strategy, I think we don’t necessarily die.
We don’t trade with ants, but we also don’t exterminate all ants. Partially this is because we can’t productively burn their bodies for energy, but actually ants are annoying and many people would like to get rid of them. It’s just not worth the effort to get rid of them, it would be going out of our way.
Similarly, an A.G.I. would see different opportunities and constraints than we do. It is a fairly defensible assumption that it would still want to grab as much negentropy and energy as possible—this underlies the same model of computation in our universe that makes me think A.G.I. will happen in the first place, so I’m not highly inclined to question it. However, it is not necessarily the case that the most efficient route to getting more energy factors through using all energy that is available locally. Burning the biosphere in Antarctica might just require too much legwork to be optimal. Once the most easily extractable energy on Earth is extracted, an A.G.I. might invest it in e.g. taking apart Mercury to build a Dyson sphere.
I guess it’s important to be clear about what I mean by “legwork” here—I expect an A.G.I. to only have so many “workers” / actuators / “arms” at a given time, and to only be able to pay attention to so many things. Perhaps it is better to focus on designing Dyson sphere technology than biosphere burning technology. I don’t see how legwork ceases to be a factor unless you believe in gray goo.
Early in A.G.I. takeover, I expect the infrastructure of humanity to be useful because an A.G.I. will not have designed radically superior infrastructure. It will probably focus on designing infrastructure for things we can’t do at all and continue to borrow our existing infrastructure for the things we are already doing because redesigning and going through the effort of replacing it isn’t the most valuable course of action on the margin.
Later in A.G.I. takeover, it will be mainly exploiting resources we don’t have access to. This probably means eating stars, but it’s possible (even likely) that our model of the incentives for Gods are not accurate. Jaynes expected less conflict between beings with radically different utility functions—obviously this is false when interpreted to mean “radically different goals for the universe” but I think it may hold up in the context he originally meant it, when different people want access to different resources. If Dyson sphere level takes awhile to achieve, and in the meantime A.G.I. has been trading with humans, human society will be much more productive, and A.G.I. will care much LESS about exploiting the remaining resources on Earth as efficiently as possible. So trade may continue at this point.
There are two obvious objections, and I think the first is persuasive and the second isn’t.
1: The A.G.I. will eventually take apart our sun which will kill us.
Continuing under the assumption that the A.G.I. cares about us literally not at all and has complete control (that is, conditioning on the story I’ve told above) I think this is very likely but still not certain. For one thing, I am only say 98% sure an A.G.I. would still care about getting more energy at this stage, and not something else we have no name for. For another, if trade really has continued with the increasingly competent society of humans on Earth, we might not need the sun to survive—we could get some batteries in exchange for doing very low priority homework sets etc., and we could use geothermal. Presumably the sun would still be eventually dissassembled, and we probably die not so long after that point (I guess we could try changing to a star the A.G.I. isn’t using), but it’s not obvious to me that even Gods eat stars in less than centuries. Maybe you get more energy out if you eat them slowly, maybe it’s just hard to eat them fast. So those of us currently alive don’t necessarily suffer untimely deaths.
2: The A.G.I. will kill us so we don’t build another A.G.I.
No need—if our society is useful it would be very easy to grab all the GPU’s from us without killing us. And the faster the takeoff, the less threat an A.G.I. created later (say, without GPU’s) would pose—whereas as slow takeoff is favorable for other reasons. I think there are actually a very narrow range of possibilities where A.G.I. kills us ONLY because it’s afraid we’ll build another A.G.I. We aren’t nearly competent enough to get away with that in real life.
(As another interesting but unrelated note on the fragility of value: though some humans may try to remove the suffering from nature, if I get a fraction of the lightcone some true wilderness will remain. So I don’t buy the example @So8res chose an I’m not sure I buy into the ultra-strong version of fragility of value.)
If AGI bulds Dyson sphere, we are dead from the simple fact of not having sunlight.
Technology of disassembling Mercury is not different from technology of disassembling Moon/Earth and easy to use to kill everyone—you just shoot relativistic projectiles using electromagnetic propulsion and evaporate swatches of planetary crust.
1: I already provided several answers to this.
2: Yes, but once Dyson sphere building tech is available I am not sure dissassembling Earth will be useful on the margin. I think Mercury provides sufficient raw materials to build a Dyson sphere and far more energy can be extracted by optimizing the Dyson sphere or hopping to other stars than grabbing the tiny amount available on Earth. Also, Earth is already home to a lot of well developed infrastructure. To the extent that takeoff looks more Hansonian than Yudkowskian, this infrastructure will become much more valuable during takeoff, and ripping it up for parts may not be wise.
My intuition is that Earth would probably be destroyed, but I think it’s worth pointing out that the economic calculation isn’t actually trivial. It seems that most rationalists expect an A.G.I. to sort of omnipotently grab all resources in the lightcone, but perhaps it would still face tradeoffs and need to prioritize—and this includes potentially pursuing opportunities we aren’t even aware of, which may not interfere with us at all.
I appreciate the speculation about this.
Such effort would most likely be a trivial expenditure compared to the resources those actions are about acquiring, and wouldn’t be as likely to entail significant opportunity costs as in the case of humans taking those actions, as AIs could parallelize their efforts when needed.
The number of Von Neumann probes one can produce should go up the more planetary material is used, so I’m not sure the adequacy of Mercury helps much. If one produces fewer probes, the expansion time (while still an exponential) starts out much slower, and at any given time growth rate would be significantly lower than it otherwise would have been.
There is a large disjunction of possible optimal behaviors, and some of these might be pursued simultaneously for the sake of avoiding risks by reserving options. Most things that look like making optimal use of resources in our solar system without considering human values are going to kill all humans.
Same, but it’d be about what portion of the sun’s output is captured, not rate of disassembly.
If this were a significant bottleneck, building new actuators or running in parallel to avoid attentional limitations would be made a high priority. I wouldn’t expect a capable AI to be significantly limited in this way for long.
An AI might not want to be highly visible to the cosmic environment and so not dim the star noticeably, or stand to get much more from acausal trade (these would still usually entail using the local resources optimally relative to those trades), or have access to negentropy stores far more vast than entailed by exploiting large celestial bodies (but what could cause the system to become fully neutral to the previously accessible resources? It would be tremendously surprising to not entail using or dissipating those resources so no competitors can arise from their use.) More energy would most likely mean earlier starts on any critical phases of its plan(s), better ability to conclude plans will work, and better ability to verify plans have worked.
True, but some parts of the situation are easier to predict than others, e.g. there’s a conjunction of many factors necessary to support human life (surface temperature as influenced by the sun / compute / atmosphere, lack of disassembly for resources, atmospheric toxicity / presence at all, strength of Earth’s magnetic field, etc), and conditioned on extreme scale unaligned AI projects that would plausibly touch many of these critical factors, the probability of survival comes out quite low for most settings of how it could go about them.
If we’re conditioning on getting an unaligned ASI, and humans are still trying to produce a friendly competitor, this seems highly likely to result in being squished. In that scenario, we’d already be conditioning on having been able to build a first AGI, so a second becomes highly probable.
The most plausible versions to me entail behaviors that either don’t look like they’re considering the presence of humans (because they don’t need to) and result in everyone dying, or are optimally exploiting the presence of humans via short-term persuasion and then repurposing humans for acausal trade scenarios or discarding them. It does seem fair to doubt we’d be given an opportunity to build a competitor, but humanity in a condition where it is unable to build AI for reasons other than foresight seems overwhelmingly likely to entail doom.
While we could be surprised by the outcome, and possibly for reasons you’ve mentioned, it still seems most probable that (given an unaligned capable AI) very capable grabbing of resources in ways that kill humans would occur, and that many rationalists are mostly working from the right model there.
I agree with most of this.
I would be modestly surprised, but not very surprised, if an A.G.I. could cause build a Dyson sphere causing the sun to be dimmed by >20% in less than a couple decades (I think a few percent isn’t enough to cause crop failure), but within a century is plausible to me.
I don’t think we would be squashed for our potential to build a competitor. I think that a competitor would no longer be a serious threat once an A.G.I. seized all available compute.
I give a little more credence to various “unknown unknowns” about the laws of physics and the priorities of superintelligences implying that an A.G.I. would no longer care to exploit the resources we need.
Overall rationalists are right to worry about being killed by A.G.I.