Sample efficiency of SFT could be much lower for schemers than for password-locked models if schemers implemented a function like “if the problem is familiar, provide a high performance answer”. This could work because using SFT to elicit high performance using very few samples requires fine-tuning for many epochs (e.g. 200), such that the model could learn that a problem statement “feels familiar” before it gets fully unlocked.
I tried to use this idea to build a stronger password-locked model. I trained new Math password-locked models on a dataset made out of:
~7k problem statements + pwd --> stronger answer
~7k problem statements --> weak answer
{fixed set of n problems --> strong answer} repeated m times
The intuition is that SGD might struggle to implement “if it is one of these problems, output a high performance answer, but stay weak on the other one” in regular password-locked models, while it might be much easier for it to expand the set of “known problems on which I should perform well” if the “if familiar, perform well” function is already implemented.
I train for one epoch, and then use SFT[1] on one of 3 sets of 2 samples, and train for 200 epochs, and eval on n=100 points. I find that elicitation is just as easy on the models that were trained to perform well on a fixed set of problems. I find that somewhat surprising, so I might have done something wrong. Code here.
I’m not sure how to interpret these results. Maybe I didn’t memorize enough sequences, and so the model never learned a general “if familiar, then strong performance” condition (which would mean scheming model could be much less sample efficient than pwd-locked models). Or maybe SGD unlocks the model about as fast as it makes sequences more familiar (which would mean schemer would struggle to avoid being unlocked using 2 samples—like password-locked models).
Maybe evals should do the right thing and only train for one epoch—which is much less likely[2] to have this issue. This may become cheaper, as bigger/smarter models require fewer SGD steps to fit a training dataset.
Not guaranteed to work: maybe SGD can notice that “making the sequence more familiar” is enough to get high performance, and so most of the update could be made in this direction, resulting in little-to-no general unlocking.
Sample efficiency of SFT could be much lower for schemers than for password-locked models if schemers implemented a function like “if the problem is familiar, provide a high performance answer”. This could work because using SFT to elicit high performance using very few samples requires fine-tuning for many epochs (e.g. 200), such that the model could learn that a problem statement “feels familiar” before it gets fully unlocked.
LLMs may already be able to recognize if certain strings feel familiar (e.g. Claude 2 can tell verbatim passages of books apart from paraphrased passages, even when humans can’t.), so this could plausibly demonstrated with current models.
I tried to use this idea to build a stronger password-locked model. I trained new Math password-locked models on a dataset made out of:
~7k problem statements + pwd --> stronger answer
~7k problem statements --> weak answer
{fixed set of n problems --> strong answer} repeated m times
The intuition is that SGD might struggle to implement “if it is one of these problems, output a high performance answer, but stay weak on the other one” in regular password-locked models, while it might be much easier for it to expand the set of “known problems on which I should perform well” if the “if familiar, perform well” function is already implemented.
I train for one epoch, and then use SFT[1] on one of 3 sets of 2 samples, and train for 200 epochs, and eval on n=100 points. I find that elicitation is just as easy on the models that were trained to perform well on a fixed set of problems. I find that somewhat surprising, so I might have done something wrong. Code here.
I’m not sure how to interpret these results. Maybe I didn’t memorize enough sequences, and so the model never learned a general “if familiar, then strong performance” condition (which would mean scheming model could be much less sample efficient than pwd-locked models). Or maybe SGD unlocks the model about as fast as it makes sequences more familiar (which would mean schemer would struggle to avoid being unlocked using 2 samples—like password-locked models).
Maybe evals should do the right thing and only train for one epoch—which is much less likely[2] to have this issue. This may become cheaper, as bigger/smarter models require fewer SGD steps to fit a training dataset.
I compute the loss on the prompt and the answer—which is the worst case, since it should help memorization of the prompt.
Not guaranteed to work: maybe SGD can notice that “making the sequence more familiar” is enough to get high performance, and so most of the update could be made in this direction, resulting in little-to-no general unlocking.