A possibly useful framing for future investigations might be, “what level of capability would an AGI need to achieve in a crucial capability in order to be dangerous”, where the definition of “dangerous” is free to vary based on how serious of a risk we are concerned about. One complication here is that this is a highly contextual question – with a superintelligence we can assume that the AGI may get basically omnipotent, but such a simplifying assumption won’t help us here. For example, the level of offensive biowarfare capability that would pose a major risk, depends on the level of the world’s defensive biowarfare capabilities. Also, we know that it’s possible to inflict enormous damage to humanity even with just human-level intelligence: whoever is authorized to control the arsenal of a nuclear power could trigger World War III, no superhuman smarts needed.
That made me properly realise something that I now feel should’ve been blindingly obvious to me already: Work to reduce humanity’s/civilization’s “vulnerabilities” in general may also help with a range of global catastrophic or existential risk scenarios where AI risk is the “trigger”.
I imagine I must’ve already been aware of that in some sense, and I think it’s implicit in various other things such as discussions of how AI could interact with e.g. nuclear weapons tech. But I don’t think I’d previously thought explicitly about situations in which risk from an agenty AI pursuing its own goals (rather than e.g. AI just making automated launch decisions) could be exacerbated or mitigated by e.g. work on biorisk, because other major risks could be what the AI would harmfully use as “tools”.
I’m not sure whether or not this should actually lead to a substantial update in e.g. how valuable I think biorisk work or work to reduce numbers of nuclear weapons is. But I’m glad to at least have that question explicitly in mind now.
This is great—thanks for posting it!
That made me properly realise something that I now feel should’ve been blindingly obvious to me already: Work to reduce humanity’s/civilization’s “vulnerabilities” in general may also help with a range of global catastrophic or existential risk scenarios where AI risk is the “trigger”.
I imagine I must’ve already been aware of that in some sense, and I think it’s implicit in various other things such as discussions of how AI could interact with e.g. nuclear weapons tech. But I don’t think I’d previously thought explicitly about situations in which risk from an agenty AI pursuing its own goals (rather than e.g. AI just making automated launch decisions) could be exacerbated or mitigated by e.g. work on biorisk, because other major risks could be what the AI would harmfully use as “tools”.
I’m not sure whether or not this should actually lead to a substantial update in e.g. how valuable I think biorisk work or work to reduce numbers of nuclear weapons is. But I’m glad to at least have that question explicitly in mind now.