engineer pandemics or conduct military operations could lead to severe derailment without satisfying this definition.
I think humans could already do those things pretty well without AI, if they wanted to. Narrow AI might make those things easier, possibly much easier, just like nukes and biotech research have in the past. I agree this increases the chance that things go “off the rails”, but I think once you have an AI that can solve hard engineering problems in the real world like that, there’s just not that much further to go to full-blown superintelligence, whether you call its precursor “narrow” or not.
The probabilities in my OP are mostly just a gut sense wild guess, but they’re based on the intuition that it takes a really big derailment to halt frontier capabilities progress, which mostly happens in well-funded labs that have the resources and will to continue operating through pretty severe “turbulence”—economic depression, war, pandemics, restrictive regulation, etc. Even if new GPU manufacturing stops completely, there are already a lot of H100s and A100s lying around, and I expect that those are sufficient to get pretty far.
I think humans could already do those things pretty well without AI, if they wanted to. Narrow AI might make those things easier, possibly much easier, just like nukes and biotech research have in the past. I agree this increases the chance that things go “off the rails”, but I think once you have an AI that can solve hard engineering problems in the real world like that, there’s just not that much further to go to full-blown superintelligence, whether you call its precursor “narrow” or not.
The probabilities in my OP are mostly just a gut sense wild guess, but they’re based on the intuition that it takes a really big derailment to halt frontier capabilities progress, which mostly happens in well-funded labs that have the resources and will to continue operating through pretty severe “turbulence”—economic depression, war, pandemics, restrictive regulation, etc. Even if new GPU manufacturing stops completely, there are already a lot of H100s and A100s lying around, and I expect that those are sufficient to get pretty far.