Glad to see public writing on what safety cases could look like!
I liked the section “Common precautions”. Points like “wherever possible, run supervised fine-tuning on the eval”, “run the eval on multiple model sizes”, “ensure that we don’t train against measurements” and “all generated outputs are monitored” are commonly acknowledged as useful/important (even if obvious), so it’s nice they were explicitly mentioned here.
In the AI control sketch, there’s a part that I find unsatisfactory:
The situations for the game were generated using Simulation Procedure S, which we are confident generates situations from a distribution that the model cannot distinguish from the genuine deployment distribution. (See below for validating the indistinguishability of this distribution.)
It’s a strong assumption that we have such a simulation procedure, at least if we are talking about General External Deployment (GED). In the Limitations section later, you write
Diversity of simulations. It seems very difficult to produce realistic simulations for a sufficiently diverse set of situations to achieve indistinguishability. Manual construction is infeasible (note that context lengths are likely to be very long). Automatically generating realistic contexts is a challenging research problem.
but I think that even this is still understating the issue: For example, during GED a user might ask a model to proofread a yet-unpublished graduate level textbook on analytic number theory she has been writing. It would be very costly to generate a diverse set of such inputs for the evaluation, and automatic generation faces the challenge of evaluation being easier than generation.
The best solution for this that I’m aware of is to train and use a trusted model to flag inputs that likely weren’t in the evaluation set. Now, you mention a related technique in “[P2.5] Optional Steps.”
Information hiding. Each input to an oversight model is first fed to a smaller, trusted model trained to check for clear signs that an input is either test or deployment. Such details (such as clear evidence of the date, or other side channels) are explicitly censored in all inputs such that the presence of censorship does not itself carry information about deployment status.
but I think this is far from enough: in the “textbook on analytic number theory” example, it’s extremely hard to hide the fact that this input would have been costly to simulate.[1]
My answer would be to have a human oversee the model’s actions on inputs flagged by the trusted model, and use such inputs to continually train and evaluate the model. But this is costly and not privacy-preserving. (The cost is acceptable if these inputs are rare enough, though. And if such inputs are common, one needs to put more effort to bridge the evaluation-deployment gap.)
I also worry about multimodality (e.g. native computer usage) making it harder to 1) make realistic simulations and 2) modify inputs in real time to hide information.
Glad to see public writing on what safety cases could look like!
I liked the section “Common precautions”. Points like “wherever possible, run supervised fine-tuning on the eval”, “run the eval on multiple model sizes”, “ensure that we don’t train against measurements” and “all generated outputs are monitored” are commonly acknowledged as useful/important (even if obvious), so it’s nice they were explicitly mentioned here.
In the AI control sketch, there’s a part that I find unsatisfactory:
It’s a strong assumption that we have such a simulation procedure, at least if we are talking about General External Deployment (GED). In the Limitations section later, you write
but I think that even this is still understating the issue: For example, during GED a user might ask a model to proofread a yet-unpublished graduate level textbook on analytic number theory she has been writing. It would be very costly to generate a diverse set of such inputs for the evaluation, and automatic generation faces the challenge of evaluation being easier than generation.
The best solution for this that I’m aware of is to train and use a trusted model to flag inputs that likely weren’t in the evaluation set. Now, you mention a related technique in “[P2.5] Optional Steps.”
but I think this is far from enough: in the “textbook on analytic number theory” example, it’s extremely hard to hide the fact that this input would have been costly to simulate.[1]
My answer would be to have a human oversee the model’s actions on inputs flagged by the trusted model, and use such inputs to continually train and evaluate the model. But this is costly and not privacy-preserving. (The cost is acceptable if these inputs are rare enough, though. And if such inputs are common, one needs to put more effort to bridge the evaluation-deployment gap.)
I also worry about multimodality (e.g. native computer usage) making it harder to 1) make realistic simulations and 2) modify inputs in real time to hide information.