Bringing technical vulnerabilities to near zero probably won’t change the AGI safety environment appreciably. It would reduce the attack surface somewhat, but that just leaves everything else. Even humans can easily induce other humans to do things that bypass security measures, and it should be presumed that a superhuman AGI can do that even better. There are also a great many unforced errors that humans make.
This sort of thing should be done as a matter of course, but it’s expensive and so far the magnitude of losses don’t make it worthwhile. Yes, software failures cost probably tens of billions of dollars per year, but building software to those standards would cost on the order of trillions of dollars per year. Fermi estimate based on verifiable software methodologies taking on the order of 5-10x longer than the usual move fast and break things processes (down from the very much more that it takes currently), and something on the order of 30 million software developers in the world that would have to be multiplied to get the same stuff.
It is more likely that if such an approach became mandatory somehow then we would develop a lot less software, but that too would cost us a great deal in terms of lost opportunities. It’s not all written for zero-sum marketing games, it just looks like it sometimes, and it wouldn’t help a great deal against superhuman AGI anyway.
Bringing technical vulnerabilities to near zero probably won’t change the AGI safety environment appreciably. It would reduce the attack surface somewhat, but that just leaves everything else. Even humans can easily induce other humans to do things that bypass security measures, and it should be presumed that a superhuman AGI can do that even better. There are also a great many unforced errors that humans make.
This sort of thing should be done as a matter of course, but it’s expensive and so far the magnitude of losses don’t make it worthwhile. Yes, software failures cost probably tens of billions of dollars per year, but building software to those standards would cost on the order of trillions of dollars per year. Fermi estimate based on verifiable software methodologies taking on the order of 5-10x longer than the usual move fast and break things processes (down from the very much more that it takes currently), and something on the order of 30 million software developers in the world that would have to be multiplied to get the same stuff.
It is more likely that if such an approach became mandatory somehow then we would develop a lot less software, but that too would cost us a great deal in terms of lost opportunities. It’s not all written for zero-sum marketing games, it just looks like it sometimes, and it wouldn’t help a great deal against superhuman AGI anyway.