The military and information assurance communities, which are used to dealing with highly adversarial environments, do not search for solutions that render all failures an impossibility.
In information security, practitioners do not look for airtight guarantees of security, but instead try to increase security iteratively as much as possible. Even RSA, the centerpiece of internet encryption, is not provably completely unbreakable (perhaps a superintelligence could find a way to efficiently factor large numbers).
I take your point, and I like the analogy to computer security. But it does seem like cryptography has had a good record of producing innovations that stem from aiming at rigorous guarantees under certain assumptions, and has been extremely valuable to improving the state of computer security.
Do you think that this claim is overstated, or just that we should additionally rely on other approaches?
That’s true! Cryptography is amenable to rigorous guarantees and proofs. I gave the example of RSA because even then, the assumption is essentially “we don’t know how to efficiently factor large numbers, and lots of people have tried, so we will assume it can’t be done efficiently.”
There are two broader points though:
Information assurance is about far more than encryption. Encryption involves proofs about algorithms, but it doesn’t guarantee information assurance (for instance, how do you keep private keys private?)
We argue in our second post that theory is limited for deep learning (in a way it is less so in cryptography). Of course, cryptography may itself be relevant to ML safety in the sense that it can be used to secure model weights, etc.
Agreed on (1) and (2). I’m still interested in the counterfactual value of theoretical research in security. One reason is that the “reasoning style” of ELK seems quite similar to that of cryptography – and at least we have some track record with the development of computer security.
I take your point, and I like the analogy to computer security. But it does seem like cryptography has had a good record of producing innovations that stem from aiming at rigorous guarantees under certain assumptions, and has been extremely valuable to improving the state of computer security.
Do you think that this claim is overstated, or just that we should additionally rely on other approaches?
That’s true! Cryptography is amenable to rigorous guarantees and proofs. I gave the example of RSA because even then, the assumption is essentially “we don’t know how to efficiently factor large numbers, and lots of people have tried, so we will assume it can’t be done efficiently.”
There are two broader points though:
Information assurance is about far more than encryption. Encryption involves proofs about algorithms, but it doesn’t guarantee information assurance (for instance, how do you keep private keys private?)
We argue in our second post that theory is limited for deep learning (in a way it is less so in cryptography). Of course, cryptography may itself be relevant to ML safety in the sense that it can be used to secure model weights, etc.
Agreed on (1) and (2). I’m still interested in the counterfactual value of theoretical research in security. One reason is that the “reasoning style” of ELK seems quite similar to that of cryptography – and at least we have some track record with the development of computer security.