Katja Grace lists a few more convergent instrumental goals that an AGI would have absent some special measure to moderate that goal. It seems to me that the usual risk from AI can be phrased as a CIV of “kill all humans”. Not just because you are made of atoms that can be used for something else, but because if our goals differ, we humans are likely to act to frustrate the goals of the AGI and even to destroy it, in order to maximize our own values; killing us all mitigates that risk.
Convergent instrumental goal: KIll All Humans
Katja Grace lists a few more convergent instrumental goals that an AGI would have absent some special measure to moderate that goal. It seems to me that the usual risk from AI can be phrased as a CIV of “kill all humans”. Not just because you are made of atoms that can be used for something else, but because if our goals differ, we humans are likely to act to frustrate the goals of the AGI and even to destroy it, in order to maximize our own values; killing us all mitigates that risk.