There’s at least two different senses in which “control” can “fail” for a powerful system:
Control evaluations can indicate that there’s no way to deploy the system such that you both (i) get a lot of use out of it, and (ii) can get a low probability of catastrophe.
Control evaluations are undermined such that humans think that the model can be deployed safely, but actually the humans were misled and there’s a high probability of catastrophe.
My impression is that Ryan & Buck typically talks about the first case. (E.g. in the link above.) I.e.: My guess would be that they’re not saying that well-designed control evaluations become untrustworthy — just that they’ll stop promising you safety.
But to be clear: In this question, you’re asking about something more analogous to the second case, right? (Sabotage/sandbagging evaluations being misleading about models’ actual capabilities at sabotage & sandbagging?)
My question posed in other words: Would you count “evaluations clearly say that models can sabotage & sandbag” as success or failure?
My guess would be that they’re not saying that well-designed control evaluations become untrustworthy
It’s a bit messy because we have some ability to check whether we should be able to evaluate things.
So, there are really three relevant “failure” states for well done control:
We can’t find countermeasures such that our control evaluations indicate any real safety.
We no longer think that control evaluations work to evaluate safety as models greatly exceed our planning capabilites and/or can sandbag sufficiently well.
We think control evaluations can work well, but we’re wrong and they actually don’t.
I think (1) or (2) will likely happen prior to (3) if you do a good job.
(Really, the actual system is more complex because we can vary how conservative control evaluations are and it is easier to be confident in the results of more conservative evaluations. E.g., we can be conservative about our AIs planning ability. So, there is really a somewhat more continuous tradeoff than the above list makes it seem.)
I basically agree with everything @Lukas Finnveden and @ryan_greenblatt said above. Another relevant “failure” state is that we might only be able to come up with expensive or capability-reducing mitigations, meaning everyone will have an incentive to disable or work around them.
There’s at least two different senses in which “control” can “fail” for a powerful system:
Control evaluations can indicate that there’s no way to deploy the system such that you both (i) get a lot of use out of it, and (ii) can get a low probability of catastrophe.
Control evaluations are undermined such that humans think that the model can be deployed safely, but actually the humans were misled and there’s a high probability of catastrophe.
My impression is that Ryan & Buck typically talks about the first case. (E.g. in the link above.) I.e.: My guess would be that they’re not saying that well-designed control evaluations become untrustworthy — just that they’ll stop promising you safety.
But to be clear: In this question, you’re asking about something more analogous to the second case, right? (Sabotage/sandbagging evaluations being misleading about models’ actual capabilities at sabotage & sandbagging?)
My question posed in other words: Would you count “evaluations clearly say that models can sabotage & sandbag” as success or failure?
It’s a bit messy because we have some ability to check whether we should be able to evaluate things.
So, there are really three relevant “failure” states for well done control:
We can’t find countermeasures such that our control evaluations indicate any real safety.
We no longer think that control evaluations work to evaluate safety as models greatly exceed our planning capabilites and/or can sandbag sufficiently well.
We think control evaluations can work well, but we’re wrong and they actually don’t.
I think (1) or (2) will likely happen prior to (3) if you do a good job.
We discuss this more here.
(Really, the actual system is more complex because we can vary how conservative control evaluations are and it is easier to be confident in the results of more conservative evaluations. E.g., we can be conservative about our AIs planning ability. So, there is really a somewhat more continuous tradeoff than the above list makes it seem.)
I basically agree with everything @Lukas Finnveden and @ryan_greenblatt said above. Another relevant “failure” state is that we might only be able to come up with expensive or capability-reducing mitigations, meaning everyone will have an incentive to disable or work around them.