Yeah, I definitely agree with “this problem doesn’t seem obviously impossible,” at least to push on quantitatively. Seems like there are a bunch of tricks from “choosing easy questions humans are confident about” to “giving the human access to AI assistants / doing debate” to “devising and testing debiasing tools” (what kinds of argument patterns are systematically more likely to convince listeners of true things rather than false things and can we train AI debaters to emulate those argument patterns?) to “asking different versions of the AI the same question and checking for consistency.” I only meant to say that the gap is big in naive HFDT, under the “naive safety effort” assumption made in the post. I think non-naive efforts will quantitatively reduce the gap in reward between honest and dishonest policies, though probably there will still be some gap in which at-least-sometimes-dishonest strategies do better than always-honest strategies. But together with other advances like interpretability or a certain type of regularization we could maybe get gradient descent to overall favor honesty.
Yeah, I definitely agree with “this problem doesn’t seem obviously impossible,” at least to push on quantitatively. Seems like there are a bunch of tricks from “choosing easy questions humans are confident about” to “giving the human access to AI assistants / doing debate” to “devising and testing debiasing tools” (what kinds of argument patterns are systematically more likely to convince listeners of true things rather than false things and can we train AI debaters to emulate those argument patterns?) to “asking different versions of the AI the same question and checking for consistency.” I only meant to say that the gap is big in naive HFDT, under the “naive safety effort” assumption made in the post. I think non-naive efforts will quantitatively reduce the gap in reward between honest and dishonest policies, though probably there will still be some gap in which at-least-sometimes-dishonest strategies do better than always-honest strategies. But together with other advances like interpretability or a certain type of regularization we could maybe get gradient descent to overall favor honesty.