It feels like privileging a hypothesis. Why should be an uFAI which ignores space travel more likely than an uFAI which ignores people dressed in green?
(The modus tollens here is that nobody would propose dressing in green as a way to reduce existential risk. And it is much cheaper than space travel.)
Preliminary note: While my assertion concerning uFAI x-risk reduction is certainly fair game for debate, it is ancillary to my main interest in this topic, which is overall x-risk reduction from all sources. That being said, I do think that the uFAI specific x-risk reduction is non-negligible, though I do agree it may well be minor.
Why should be an uFAI which ignores space travel more likely than an uFAI which ignores people dressed in green?
Two broad categories of explaining such a difference:
The advent of uFAI may lead to a series of events (e.g. nuclear winter) that 1) preclude the uFAI from pursuing space travel for the time being, 2) lead to the mutual demise of both humankind and the uFAI or 3) lead to a situation in which the cost/benefit analysis on the uFAI’s part does not come out in favor of wiping out a Mars colony or 4) leaves the Mars colony enough time to implement counter measures of some sort, up to and including creating a friendly AI to protect them.
The utility function (which may well be a somewhat random one implemented by a researcher unwittingly creating the first AGI) could well yield strange results, especially if it is not change-invariant. For example, it may have an emphasis on building tools unable to achieve space flight (maybe the uFAI was originally supposed to only build as many cars as possible, favoring certain tools), be only concerned with the planet earth (“Save the planet”-type AI) or—as mentioned—be incapable of pursuing long-term plans due to geometric discounting of future rewards, and there always being something to optimize which only takes short-term planning (i.e. locked in a greedy pattern).
All of which is of course speculative, but a uFAI taking to the stars has “more”* scenarios going against it than a uFAI ignoring people dressed in green. (* in terms of composite probability, the number of scenarios is countably infinite for both)
OK, makes sense. If we assume the AI to be perfectly rational, it would probably give exterminating humanity out of Earth high priority, exactly because there is a chance of them building another AI.
However, to wipe out humanity from the Earth, the AI does not have to be very smart. One virus, well designed and well distributed, could do the job. An AI with some bugs could still be capable to make it… and then fail to properly arrange the space attack, or destroy itself by wrong self-modification.
The set of UFAI that threaten people dressed in green is strictly smaller than the set of all UFAI as well, technically. However, for all practical purposes the sets can be treated as just identical.
It feels like privileging a hypothesis. Why should be an uFAI which ignores space travel more likely than an uFAI which ignores people dressed in green?
(The modus tollens here is that nobody would propose dressing in green as a way to reduce existential risk. And it is much cheaper than space travel.)
Preliminary note: While my assertion concerning uFAI x-risk reduction is certainly fair game for debate, it is ancillary to my main interest in this topic, which is overall x-risk reduction from all sources. That being said, I do think that the uFAI specific x-risk reduction is non-negligible, though I do agree it may well be minor.
Two broad categories of explaining such a difference:
The advent of uFAI may lead to a series of events (e.g. nuclear winter) that 1) preclude the uFAI from pursuing space travel for the time being, 2) lead to the mutual demise of both humankind and the uFAI or 3) lead to a situation in which the cost/benefit analysis on the uFAI’s part does not come out in favor of wiping out a Mars colony or 4) leaves the Mars colony enough time to implement counter measures of some sort, up to and including creating a friendly AI to protect them.
The utility function (which may well be a somewhat random one implemented by a researcher unwittingly creating the first AGI) could well yield strange results, especially if it is not change-invariant. For example, it may have an emphasis on building tools unable to achieve space flight (maybe the uFAI was originally supposed to only build as many cars as possible, favoring certain tools), be only concerned with the planet earth (“Save the planet”-type AI) or—as mentioned—be incapable of pursuing long-term plans due to geometric discounting of future rewards, and there always being something to optimize which only takes short-term planning (i.e. locked in a greedy pattern).
All of which is of course speculative, but a uFAI taking to the stars has “more”* scenarios going against it than a uFAI ignoring people dressed in green. (* in terms of composite probability, the number of scenarios is countably infinite for both)
OK, makes sense. If we assume the AI to be perfectly rational, it would probably give exterminating humanity out of Earth high priority, exactly because there is a chance of them building another AI.
However, to wipe out humanity from the Earth, the AI does not have to be very smart. One virus, well designed and well distributed, could do the job. An AI with some bugs could still be capable to make it… and then fail to properly arrange the space attack, or destroy itself by wrong self-modification.
The set of UFAI that threaten people dressed in green is strictly smaller than the set of all UFAI as well, technically. However, for all practical purposes the sets can be treated as just identical.