This should also at least somewhat reduce the x-risk stemming from uFAI.
I doubt it. If travel to Mars becomes do-able for Musk, it’ll be trivial for uFAI, and uFAI wouldn’t be stupid enough to let a nearby technological civilization threaten the efficient achievement of its goals, whatever they are.
Correct. SIs that only terminally care about a single green button on Earth instrumentally care about optimizing the rest of the universe to prevent anything from threatening that button.
Good point, and this appears to be a general issue with human space settlement.
Suppose that the technology to do it cost-effectively arrives before AI e.g. cheap spaceflight arrives this century, and AI doesn’t. Then even if an unfriendly AI shows up centuries later, it can catch-up with and overwhelm the human settlements, no matter how far from Earth they’ve reached. (The AI can survive higher accelerations, reach higher speeds, reproduce more quickly before launching new settlement missions etc.)
Worse, increasing the number of human settlements most likely increases the chance that someone, somewhere builds an unfriendly AI. So, somewhat surprisingly, space settlement could increase existential risk rather than reduce it.
Worse, increasing the number of human settlements most likely increases the chance that someone, somewhere builds an unfriendly AI.
I don’t see this logic. The chance of AI being made is based on how many people are working on it, not how many locations there are where people can work on AI.
More extreme environments may force faster technological developments ( eg WW2), also see the excellent “Destination Void” by Frank Herbert, where a colony ship has to develop an AI to survive the journey to the nearest star, with unexpected results!
I believe the reasoning is [more human settlements] --> [more total humans] --> [more humans to make an AI]. Whether or not settlements on eg Mars will actually lead to more total humans on the timescale we’re talking about is up for debate.
Yes, there is this effect. I was making a general point about what happens if we get space settlements a long time between AI (or to be precise AGI). More lebensraum gives more people.
Also, the number of remote populations increases the chance of someone building an unfriendly AI. A single Earth-bound community has more chance of co-ordinating research on AI safety (which includes slowing down AI deployment until they are confident about safety, and policing rogue effects that just want to get on with building AIs). This seems much harder to achieve if there are lots of groups spread widely apart, with long travel times between them, and no interplanetary cops forcing everyone to behave. Or there may be fierce competition, with everyone wanting to get their own (presumed safe) AIs deployed before nasty ones show up from the other planets.
In many cases, spaceflight may not even be necessary, an uFAI that is still in stealth mode could just transmit itself over.
However (correct me on the jargon), uFAI mindspace certainly contains uFAI’s that—for whatever reason—will not concern themselves with anything other than Earth.
For example, a uFAI that uses geometric discounting as part of its utility function (cf. page 6 here) may realize the eventual danger (decades down the line) from a Mars colony, but may invest all its resources to transform Earth into the desired state anyways.
Since the number of uFAI concepts that will also threaten Mars is strictly smaller (even if not much so) than the number of uFAI concepts with no constraints on Mars given, I think the statement that the risk would be somewhat smaller is fair to make (conjunction fallacy otherwise).
It feels like privileging a hypothesis. Why should be an uFAI which ignores space travel more likely than an uFAI which ignores people dressed in green?
(The modus tollens here is that nobody would propose dressing in green as a way to reduce existential risk. And it is much cheaper than space travel.)
Preliminary note: While my assertion concerning uFAI x-risk reduction is certainly fair game for debate, it is ancillary to my main interest in this topic, which is overall x-risk reduction from all sources. That being said, I do think that the uFAI specific x-risk reduction is non-negligible, though I do agree it may well be minor.
Why should be an uFAI which ignores space travel more likely than an uFAI which ignores people dressed in green?
Two broad categories of explaining such a difference:
The advent of uFAI may lead to a series of events (e.g. nuclear winter) that 1) preclude the uFAI from pursuing space travel for the time being, 2) lead to the mutual demise of both humankind and the uFAI or 3) lead to a situation in which the cost/benefit analysis on the uFAI’s part does not come out in favor of wiping out a Mars colony or 4) leaves the Mars colony enough time to implement counter measures of some sort, up to and including creating a friendly AI to protect them.
The utility function (which may well be a somewhat random one implemented by a researcher unwittingly creating the first AGI) could well yield strange results, especially if it is not change-invariant. For example, it may have an emphasis on building tools unable to achieve space flight (maybe the uFAI was originally supposed to only build as many cars as possible, favoring certain tools), be only concerned with the planet earth (“Save the planet”-type AI) or—as mentioned—be incapable of pursuing long-term plans due to geometric discounting of future rewards, and there always being something to optimize which only takes short-term planning (i.e. locked in a greedy pattern).
All of which is of course speculative, but a uFAI taking to the stars has “more”* scenarios going against it than a uFAI ignoring people dressed in green. (* in terms of composite probability, the number of scenarios is countably infinite for both)
OK, makes sense. If we assume the AI to be perfectly rational, it would probably give exterminating humanity out of Earth high priority, exactly because there is a chance of them building another AI.
However, to wipe out humanity from the Earth, the AI does not have to be very smart. One virus, well designed and well distributed, could do the job. An AI with some bugs could still be capable to make it… and then fail to properly arrange the space attack, or destroy itself by wrong self-modification.
The set of UFAI that threaten people dressed in green is strictly smaller than the set of all UFAI as well, technically. However, for all practical purposes the sets can be treated as just identical.
I doubt it. If travel to Mars becomes do-able for Musk, it’ll be trivial for uFAI, and uFAI wouldn’t be stupid enough to let a nearby technological civilization threaten the efficient achievement of its goals, whatever they are.
Correct. SIs that only terminally care about a single green button on Earth instrumentally care about optimizing the rest of the universe to prevent anything from threatening that button.
Good point, and this appears to be a general issue with human space settlement.
Suppose that the technology to do it cost-effectively arrives before AI e.g. cheap spaceflight arrives this century, and AI doesn’t. Then even if an unfriendly AI shows up centuries later, it can catch-up with and overwhelm the human settlements, no matter how far from Earth they’ve reached. (The AI can survive higher accelerations, reach higher speeds, reproduce more quickly before launching new settlement missions etc.)
Worse, increasing the number of human settlements most likely increases the chance that someone, somewhere builds an unfriendly AI. So, somewhat surprisingly, space settlement could increase existential risk rather than reduce it.
I don’t see this logic. The chance of AI being made is based on how many people are working on it, not how many locations there are where people can work on AI.
More extreme environments may force faster technological developments ( eg WW2), also see the excellent “Destination Void” by Frank Herbert, where a colony ship has to develop an AI to survive the journey to the nearest star, with unexpected results!
I believe the reasoning is [more human settlements] --> [more total humans] --> [more humans to make an AI]. Whether or not settlements on eg Mars will actually lead to more total humans on the timescale we’re talking about is up for debate.
Yes, there is this effect. I was making a general point about what happens if we get space settlements a long time between AI (or to be precise AGI). More lebensraum gives more people.
Also, the number of remote populations increases the chance of someone building an unfriendly AI. A single Earth-bound community has more chance of co-ordinating research on AI safety (which includes slowing down AI deployment until they are confident about safety, and policing rogue effects that just want to get on with building AIs). This seems much harder to achieve if there are lots of groups spread widely apart, with long travel times between them, and no interplanetary cops forcing everyone to behave. Or there may be fierce competition, with everyone wanting to get their own (presumed safe) AIs deployed before nasty ones show up from the other planets.
In many cases, spaceflight may not even be necessary, an uFAI that is still in stealth mode could just transmit itself over.
However (correct me on the jargon), uFAI mindspace certainly contains uFAI’s that—for whatever reason—will not concern themselves with anything other than Earth.
For example, a uFAI that uses geometric discounting as part of its utility function (cf. page 6 here) may realize the eventual danger (decades down the line) from a Mars colony, but may invest all its resources to transform Earth into the desired state anyways.
Since the number of uFAI concepts that will also threaten Mars is strictly smaller (even if not much so) than the number of uFAI concepts with no constraints on Mars given, I think the statement that the risk would be somewhat smaller is fair to make (conjunction fallacy otherwise).
It feels like privileging a hypothesis. Why should be an uFAI which ignores space travel more likely than an uFAI which ignores people dressed in green?
(The modus tollens here is that nobody would propose dressing in green as a way to reduce existential risk. And it is much cheaper than space travel.)
Preliminary note: While my assertion concerning uFAI x-risk reduction is certainly fair game for debate, it is ancillary to my main interest in this topic, which is overall x-risk reduction from all sources. That being said, I do think that the uFAI specific x-risk reduction is non-negligible, though I do agree it may well be minor.
Two broad categories of explaining such a difference:
The advent of uFAI may lead to a series of events (e.g. nuclear winter) that 1) preclude the uFAI from pursuing space travel for the time being, 2) lead to the mutual demise of both humankind and the uFAI or 3) lead to a situation in which the cost/benefit analysis on the uFAI’s part does not come out in favor of wiping out a Mars colony or 4) leaves the Mars colony enough time to implement counter measures of some sort, up to and including creating a friendly AI to protect them.
The utility function (which may well be a somewhat random one implemented by a researcher unwittingly creating the first AGI) could well yield strange results, especially if it is not change-invariant. For example, it may have an emphasis on building tools unable to achieve space flight (maybe the uFAI was originally supposed to only build as many cars as possible, favoring certain tools), be only concerned with the planet earth (“Save the planet”-type AI) or—as mentioned—be incapable of pursuing long-term plans due to geometric discounting of future rewards, and there always being something to optimize which only takes short-term planning (i.e. locked in a greedy pattern).
All of which is of course speculative, but a uFAI taking to the stars has “more”* scenarios going against it than a uFAI ignoring people dressed in green. (* in terms of composite probability, the number of scenarios is countably infinite for both)
OK, makes sense. If we assume the AI to be perfectly rational, it would probably give exterminating humanity out of Earth high priority, exactly because there is a chance of them building another AI.
However, to wipe out humanity from the Earth, the AI does not have to be very smart. One virus, well designed and well distributed, could do the job. An AI with some bugs could still be capable to make it… and then fail to properly arrange the space attack, or destroy itself by wrong self-modification.
The set of UFAI that threaten people dressed in green is strictly smaller than the set of all UFAI as well, technically. However, for all practical purposes the sets can be treated as just identical.
somewhat = epsilon