Eliezer, whenever you start thinking about people who are completely causally unconnected with us as morally relevant, alarm bells should go off.
What’s worse though, is that if your opinion on this is driven by a desire to justify not agreeing with the “repugnant conclusion”, it may signify problems with your morality that could annihilate humanity if you give your morality to an AI. The repugnant conclusion requires valuing the bringing into existence of hypothetical people with total utility x by as much as reducing the utility of existing people by x, or annihilating people with utility x. Give that morality to a fast takeoff AI and they’ll quickly replace all humans with entities with greater capacity for utility. If the AI is programmed to believe the problem with the “repugnant conclusion” is what you claim, the AI will instead create randomized (for high uniqueness) minds with high capacity for utility, still annihilating humans.
Eliezer, whenever you start thinking about people who are completely causally unconnected with us as morally relevant, alarm bells should go off.
What’s worse though, is that if your opinion on this is driven by a desire to justify not agreeing with the “repugnant conclusion”, it may signify problems with your morality that could annihilate humanity if you give your morality to an AI. The repugnant conclusion requires valuing the bringing into existence of hypothetical people with total utility x by as much as reducing the utility of existing people by x, or annihilating people with utility x. Give that morality to a fast takeoff AI and they’ll quickly replace all humans with entities with greater capacity for utility. If the AI is programmed to believe the problem with the “repugnant conclusion” is what you claim, the AI will instead create randomized (for high uniqueness) minds with high capacity for utility, still annihilating humans.