Even with powerful AI assistance, a pretty confident lower bound on what’s required to achieve this seems like a 5-10-year Manhattan project; so I expect most of this wouldn’t happen in time for Aschenbrenner-like timelines, even with significant resources poured in.
Timelines are certainly short. But if AI systems progress as rapidly as Aschenbrenner and others are predicting, then most of the work will be done by AIs. Of course, we have to know what we want them to accomplish! And that’s something we can work on before the AIs are dangerous or powerful enough to build the safe infrastructure.
Max and I argued that systems which are not built with security proofs will likely have vulnerabilities and that powerful AI’s can search form them using formal models and are likely to find them. So if things unfold in an adversarial way, any software, hardware, or social systems not built with provable safety will likely be exploited.
I buy the automation argument, I don’t expect that to be the (main) bottleneck under short timelines. My worries are about needing to change the behavior of humans (e.g. programmers starting to integrate formal verification, etc.) and to upgrade large parts of existing infrastructure, especially when it’s physical.
I agree that the human responses to all of this are the great unknown. The good news is that human programmers don’t need to learn to do formal verification, they just need to realize that it’s important and to get the AIs to do it. My suspicion is that these big shifts won’t happen until we start having AI-caused disasters. It’s interesting to look at the human response to the CrowdStrike disaster and other recent security issues to get some sense of how the responses might go. It’s also interesting to see the unbelievable speed with which Microsoft, Meta, Google, etc. have been building huge new GPU datacenters in response to the projected advances in AI.
Even with powerful AI assistance, a pretty confident lower bound on what’s required to achieve this seems like a 5-10-year Manhattan project; so I expect most of this wouldn’t happen in time for Aschenbrenner-like timelines, even with significant resources poured in.
Timelines are certainly short. But if AI systems progress as rapidly as Aschenbrenner and others are predicting, then most of the work will be done by AIs. Of course, we have to know what we want them to accomplish! And that’s something we can work on before the AIs are dangerous or powerful enough to build the safe infrastructure.
Max and I argued that systems which are not built with security proofs will likely have vulnerabilities and that powerful AI’s can search form them using formal models and are likely to find them. So if things unfold in an adversarial way, any software, hardware, or social systems not built with provable safety will likely be exploited.
I buy the automation argument, I don’t expect that to be the (main) bottleneck under short timelines. My worries are about needing to change the behavior of humans (e.g. programmers starting to integrate formal verification, etc.) and to upgrade large parts of existing infrastructure, especially when it’s physical.
I agree that the human responses to all of this are the great unknown. The good news is that human programmers don’t need to learn to do formal verification, they just need to realize that it’s important and to get the AIs to do it. My suspicion is that these big shifts won’t happen until we start having AI-caused disasters. It’s interesting to look at the human response to the CrowdStrike disaster and other recent security issues to get some sense of how the responses might go. It’s also interesting to see the unbelievable speed with which Microsoft, Meta, Google, etc. have been building huge new GPU datacenters in response to the projected advances in AI.