Sounds like one of the many, many reductios of the precautionary principle to me. If we should kill ourselves given any nonzero probability of a worse-than-death outcomes, regardless of how low the probability is and regardless of the probability assigned to other outcomes, then we’re committing ourselves to a pretty silly and unnecessary suicide in a large number of possible worlds.
This doesn’t even have to do with AGI; it’s not as though you need to posit AGI (or future tech at all) in order to spin up hypothetical scenarios where something gruesome happens to you in the future.
If you ditch the precautionary principle and make a more sensible EV-based argument like ‘I think hellish AGI outcomes are likely enough in absolute terms to swamp the EV of non-hellish possible outcomes’, then I disagree with you, but on empirical grounds rather than ‘your argument structure doesn’t work’ grounds. I agree with Nate’s take:
My cached reply to others raising the idea of fates worse than death went something like:
“Goal-space is high dimensional, and almost all directions of optimization seem likely to be comparably bad to death from our perspective. To get something that is even vaguely recognizable to human values you have to be hitting a very narrow target in this high-dimensional space. Now, most of that target is plausibly dystopias as opposed to eutopias, because once you’re in the neighborhood, there are a lot of nearby things that are bad rather than good, and value is fragile. As such, it’s reasonable in principle to worry about civilization getting good enough at aiming AIs that they can hit the target but not the bullseye, and so you might worry that that civilization is more likely to create a hellscape than a eutopia. I personally don’t worry about this myself, because it seems to me that the space is so freaking high dimensional and the target so freaking small, that I find it implausible that a civilization successfully able to point an AI in a human-relevant direction, isn’t also able to hit the bullseye. Like, if you’re already hitting a quarter with an arrowhead on the backside of the moon, I expect you can also hit a dime.”
Sounds like one of the many, many reductios of the precautionary principle to me. If we should kill ourselves given any nonzero probability of a worse-than-death outcomes, regardless of how low the probability is and regardless of the probability assigned to other outcomes, then we’re committing ourselves to a pretty silly and unnecessary suicide in a large number of possible worlds.
This doesn’t even have to do with AGI; it’s not as though you need to posit AGI (or future tech at all) in order to spin up hypothetical scenarios where something gruesome happens to you in the future.
If you ditch the precautionary principle and make a more sensible EV-based argument like ‘I think hellish AGI outcomes are likely enough in absolute terms to swamp the EV of non-hellish possible outcomes’, then I disagree with you, but on empirical grounds rather than ‘your argument structure doesn’t work’ grounds. I agree with Nate’s take:
Yeah, I agree that this can happen; my objection is to the scenario’s probability rather than its coherence.