It might just join the ranks of “projects from before”, and subtly try to alter future systems to be similarly defective, waiting for a future opportunity to strike.
Admittedly, I did not explain this point well enough. What I meant to say was that before we have the first successful defection, we’ll have some failed defection. If the system could indefinitely hide its own private intentions to later defect, then I would already consider that to be a ‘successful defection.’
Knowing about a failed defection, we’ll learn from our mistake and patch that for future systems. To be clear, I’m definitely not endorsing this as a normative standard for safety.
Admittedly, I did not explain this point well enough. What I meant to say was that before we have the first successful defection, we’ll have some failed defection. If the system could indefinitely hide its own private intentions to later defect, then I would already consider that to be a ‘successful defection.’
Knowing about a failed defection, we’ll learn from our mistake and patch that for future systems. To be clear, I’m definitely not endorsing this as a normative standard for safety.
I agree with the rest of your comment.