that producing safe AI requires (what is currently) extraordinary success at “getting things right first time”, so that
See my reply above—this line of thought is fundamentally mistaken. Simulation testing is far more of an effective general solution than formal verification.
Just to clarify: I was stating the thesis of the OP, not asserting it. Neither am I now denying it. (I don’t find myself altogether convinced either by the OP or by your arguments for why the OP is “probably completely mistaken”.)
See my reply above—this line of thought is fundamentally mistaken. Simulation testing is far more of an effective general solution than formal verification.
Just to clarify: I was stating the thesis of the OP, not asserting it. Neither am I now denying it. (I don’t find myself altogether convinced either by the OP or by your arguments for why the OP is “probably completely mistaken”.)