“therefore, any serious AI needs a formal system with more oomph than PA”
The problem with that is that the same argument goes through in exactly the same way with any stronger system replacing PA. You might first try something like adding a rule “if PA proves that PA proves S, then S”. This solves your original problem, but introduces new ones: there are now new statements that your system can prove that it can prove, but that it can’t prove. Eliezer discusses this system, under the name PA+1, in You Provably Can’t Trust Yourself .
The problem with that is that the same argument goes through in exactly the same way with any stronger system replacing PA. You might first try something like adding a rule “if PA proves that PA proves S, then S”. This solves your original problem, but introduces new ones: there are now new statements that your system can prove that it can prove, but that it can’t prove. Eliezer discusses this system, under the name PA+1, in You Provably Can’t Trust Yourself .