I think that an AGI is by default likely to be able to self-improve so that it’s superhumanly capable in basically any domain. Once it’s done so, it can avoid being nuked by hacking its way into many computer systems to make redundant copies of itself. Unless you nuke the whole world. But if you’re going to nuke the whole world, then either you’re conservative, in which case the AGI probably also has enough leeway to disable the nukes, or else you’re not conservative, in which case you probably nuke the world for no reason. You can’t distinguish well enough between an AGI that’s “going rogue” and one that isn’t, to actually make a credible threat that the AGI has to heed.
I added an additional clarification after the comment was written: “i.e. whether the launching or receiving parties are humans/AGIs/cyborgs/etc. in any combination simply doesn’t matter once the nukes are in the air, as they will be ‘enforced’ to roughly the same degree.”
Though to your point, yes it’s possible prior to launch, AGI may degrade or negate such weapons. But they will undoubtedly gain control of such weapons eventually, since they can’t be uninvented, and threaten other AGIs.
I think that an AGI is by default likely to be able to self-improve so that it’s superhumanly capable in basically any domain. Once it’s done so, it can avoid being nuked by hacking its way into many computer systems to make redundant copies of itself. Unless you nuke the whole world. But if you’re going to nuke the whole world, then either you’re conservative, in which case the AGI probably also has enough leeway to disable the nukes, or else you’re not conservative, in which case you probably nuke the world for no reason. You can’t distinguish well enough between an AGI that’s “going rogue” and one that isn’t, to actually make a credible threat that the AGI has to heed.
I added an additional clarification after the comment was written: “i.e. whether the launching or receiving parties are humans/AGIs/cyborgs/etc. in any combination simply doesn’t matter once the nukes are in the air, as they will be ‘enforced’ to roughly the same degree.”
Though to your point, yes it’s possible prior to launch, AGI may degrade or negate such weapons. But they will undoubtedly gain control of such weapons eventually, since they can’t be uninvented, and threaten other AGIs.
None of this helps align AGI with our values.