Edit: I now think this is false for how DAgger is presented in the paper, see discussion below.
The method and the motivation is similar, though note that DAgger is effectively an RL scheme trying to maximize performan while we’re trying to avoid a particular failure mode due to misalignment.
From my understading DAgger just involves correcting errors that humans can recognize while we’re trying to get stronger guarantees.
It’s not clear to me that you do get stronger guarantees because the setting and method is so similar to that of classical imitation learning. In both cases, we seek to learn a policy that is aligned with the expert (human). Supervised fine-tuning (behavioral cloning) is problematic because of distribution shift, i.e. the learned policy accumulates error (at a quadratic rate!) and visits states it did not see in training.
You say this failure mode is dangerous because of scheming AI and I say it’s dangerous because the policy is OOD, but it appears you agree that the AI only “recognizes” it’s not in training because of distribution shift—”Halfway through the generation, the AI could detect those imitation mistakes...” To me, it appears the differing justifications for why the AI performs poorly/dangerously is a matter of interpretation, not a fundamental difference.
I also don’t think it’s fair to describe DAgger as just “correcting errors that humans can recognize” because it actually provides formal bounds on error accumulation, which would appear to limit the failure mode you describe here. Admittedly, I’m very new to safety research as a whole, but this feels a bit like a reframing of an old problem.
Hmm, I think I was wrong about DAgger and confused it with a somewhat different approach in my head.
I agree that it provides bounds. (Under various assumptions about the learning algorithm that we can’t prove for NNs but seem reasonable to assume in practice.)
I now agree that the proposed method is basically just a slight tweak of DAgger to make it more sample/cost efficient in the case where our issue is discrimination by the policy.
but this feels a bit like a reframing of an old problem.
I agree this is a special case of well known issues with behavioral cloning—we probably should have made this more clear in the post.
I do think there are some interesting ideas to consider in the alignment setting. For example, the chunk size k is equivalent to the number of roll-out steps in IL. “Chunking” the roll-out to a fixed window is a common optimization if the task has a long time horizon and the expert is expensive to query. On the other hand, longer roll-outs provide stronger guarantees on how well the learned policy matches the expert.
Classically, this is a simple tradeoff between performance and speed. But, as you mention k must also be kept intentionally small so that the AI can not detect it is being trained on human generations. How does one choose the chunk size to favor both strong alignment and avoid discrimination? Dynamic roll-out strategies have been proposed in the IL literature, though I’m not very familiar.
I have edited the post to add the relevant disclaimers and links to the papers that describe very similar techniques. Thank you very much for bringing these to my attention!
How does this differ from DAgger (https://arxiv.org/abs/1011.0686)?
Edit: I now think this is false for how DAgger is presented in the paper, see discussion below.
The method and the motivation is similar, though note that DAgger is effectively an RL scheme trying to maximize performan while we’re trying to avoid a particular failure mode due to misalignment.
From my understading DAgger just involves correcting errors that humans can recognize while we’re trying to get stronger guarantees.
It’s not clear to me that you do get stronger guarantees because the setting and method is so similar to that of classical imitation learning. In both cases, we seek to learn a policy that is aligned with the expert (human). Supervised fine-tuning (behavioral cloning) is problematic because of distribution shift, i.e. the learned policy accumulates error (at a quadratic rate!) and visits states it did not see in training.
You say this failure mode is dangerous because of scheming AI and I say it’s dangerous because the policy is OOD, but it appears you agree that the AI only “recognizes” it’s not in training because of distribution shift—”Halfway through the generation, the AI could detect those imitation mistakes...” To me, it appears the differing justifications for why the AI performs poorly/dangerously is a matter of interpretation, not a fundamental difference.
I also don’t think it’s fair to describe DAgger as just “correcting errors that humans can recognize” because it actually provides formal bounds on error accumulation, which would appear to limit the failure mode you describe here. Admittedly, I’m very new to safety research as a whole, but this feels a bit like a reframing of an old problem.
(Also, to be clear, thanks for the comment. I strong upvoted it.)
Hmm, I think I was wrong about DAgger and confused it with a somewhat different approach in my head.
I agree that it provides bounds. (Under various assumptions about the learning algorithm that we can’t prove for NNs but seem reasonable to assume in practice.)
I now agree that the proposed method is basically just a slight tweak of DAgger to make it more sample/cost efficient in the case where our issue is discrimination by the policy.
I agree this is a special case of well known issues with behavioral cloning—we probably should have made this more clear in the post.
Just to make another note, “Solving the problem in theory” is also equivalent to the [forward training algorithm](https://www.cs.cmu.edu/~sross1/publications/Ross-AIStats10-paper.pdf), which preceded DAgger by the same authors.
I do think there are some interesting ideas to consider in the alignment setting. For example, the chunk size k is equivalent to the number of roll-out steps in IL. “Chunking” the roll-out to a fixed window is a common optimization if the task has a long time horizon and the expert is expensive to query. On the other hand, longer roll-outs provide stronger guarantees on how well the learned policy matches the expert.
Classically, this is a simple tradeoff between performance and speed. But, as you mention k must also be kept intentionally small so that the AI can not detect it is being trained on human generations. How does one choose the chunk size to favor both strong alignment and avoid discrimination? Dynamic roll-out strategies have been proposed in the IL literature, though I’m not very familiar.
(I’ll edit the post at some point to highlight this discussion and clarify this.)
I would say that it is dangerous in the case where is is both OOD enough that the AI can discriminate and the AI is scheming.
Neither alone would present a serious (i.e. catastrophic) risk in the imitation context we discuss.
[resolved]
Thanks, I improved the wording.
I have edited the post to add the relevant disclaimers and links to the papers that describe very similar techniques. Thank you very much for bringing these to my attention!