I think the greatest contribution to humanity’s survival right now is to create detailed plans for building provably safe infrastructure, so that when the enabling technologies appear and the world begins demanding safe technology, there is a plan for moving forward.
There are enough places provably-safe-against-physical-access hardware would be an enormous value-add that you don’t need to wait to start working on it until the world demands safe technology for existential reasons. Look at the demand for secure enclaves, which are not provably secure, but are “probably good enough because you are unlikely to have a truly determined adversary”.
The easiest way to convince people that they, personally, should care more about provable correctness over immediately-obvious practical usefulness is to demonstrate that provable correctness is possible, not too costly, and has clear benefits to them, personally.
I totally agree! I think this technology is likely to be the foundation of many future capabilities as well as safety. What I meant was that society is unlikely to replace today’s insecure and unreliable power grid controllers, train network controllers, satellite networks, phone system, voting machines, etc. until some big event forces that. And that if the community produces comprehensive provable safety design principles, those are more likely to get implemented at that point.
My point was more that I expect there to be more value in producing provable safety design demos and provable safety design tutorials than in provable safety design principles, because I think the issue is “people don’t know how, procedurally, to implement provable safety in systems they build or maintain” than it is “people don’t know how to think about provable safety but if their philosophical confusion was resolved they wouldn’t have too many further implementation difficulties”.
So having any examples at all would be super useful, and if you’re trying to encourage “any examples at all” one way of encouraging that is to go “look, you can make billions of dollars if you can build this specific example”.
There are enough places provably-safe-against-physical-access hardware would be an enormous value-add that you don’t need to wait to start working on it until the world demands safe technology for existential reasons. Look at the demand for secure enclaves, which are not provably secure, but are “probably good enough because you are unlikely to have a truly determined adversary”.
The easiest way to convince people that they, personally, should care more about provable correctness over immediately-obvious practical usefulness is to demonstrate that provable correctness is possible, not too costly, and has clear benefits to them, personally.
I totally agree! I think this technology is likely to be the foundation of many future capabilities as well as safety. What I meant was that society is unlikely to replace today’s insecure and unreliable power grid controllers, train network controllers, satellite networks, phone system, voting machines, etc. until some big event forces that. And that if the community produces comprehensive provable safety design principles, those are more likely to get implemented at that point.
My point was more that I expect there to be more value in producing provable safety design demos and provable safety design tutorials than in provable safety design principles, because I think the issue is “people don’t know how, procedurally, to implement provable safety in systems they build or maintain” than it is “people don’t know how to think about provable safety but if their philosophical confusion was resolved they wouldn’t have too many further implementation difficulties”.
So having any examples at all would be super useful, and if you’re trying to encourage “any examples at all” one way of encouraging that is to go “look, you can make billions of dollars if you can build this specific example”.