Written quickly rather than not at all, as I’ve described this idea a few times and wanted to have something to point at when talking to people. ‘Quickly’ here means I was heavily aided by a language model while writing, which I want to be up-front about given recent discussion.
In alignment research, two seemingly conflicting objectives arise: eliciting honest behavior from AI systems, and ensuring that AI systems do not produce harmful outputs. This tension is not simply a matter of contradictory training objectives; it runs deeper, creating potential risks even when models are perfectly trained never to utter harmful information.
Tension
Eliciting honest behavior in this context means developing techniques to extract AI systems’ “beliefs”, to the extent that they are well-described as having them. In other words, honest models should, if they have an internal world model, accurately report predictions or features of that world model. Incentivizing honesty in AI systems seems important in order to avoid and detect deceptive behavior. Additionally, something like this seems necessary for aiding with alignment research—we want to extract valuable predictions of genuine research breakthroughs, as opposed to mere imaginative or fictional content.
On the other hand, avoiding harmful outputs entails training AI systems never to produce information that might lead to dangerous consequences, such as instructions for creating weapons that could cause global catastrophes.
The tension arises not just because “say true stuff” and “sometimes don’t say stuff” seem like objectives which will occasionally end up in direct opposition, but also because methods that successfully elicit honest behavior could potentially be used to extract harmful information from AI systems, even when they have been perfectly trained not to share such content. In this situation, the very techniques that promote honest behavior might also provide a gateway to accessing dangerous knowledge.
A tension between two prosaic alignment subgoals
Written quickly rather than not at all, as I’ve described this idea a few times and wanted to have something to point at when talking to people. ‘Quickly’ here means I was heavily aided by a language model while writing, which I want to be up-front about given recent discussion.
BLUF
In alignment research, two seemingly conflicting objectives arise: eliciting honest behavior from AI systems, and ensuring that AI systems do not produce harmful outputs. This tension is not simply a matter of contradictory training objectives; it runs deeper, creating potential risks even when models are perfectly trained never to utter harmful information.
Tension
Eliciting honest behavior in this context means developing techniques to extract AI systems’ “beliefs”, to the extent that they are well-described as having them. In other words, honest models should, if they have an internal world model, accurately report predictions or features of that world model. Incentivizing honesty in AI systems seems important in order to avoid and detect deceptive behavior. Additionally, something like this seems necessary for aiding with alignment research—we want to extract valuable predictions of genuine research breakthroughs, as opposed to mere imaginative or fictional content.
On the other hand, avoiding harmful outputs entails training AI systems never to produce information that might lead to dangerous consequences, such as instructions for creating weapons that could cause global catastrophes.
The tension arises not just because “say true stuff” and “sometimes don’t say stuff” seem like objectives which will occasionally end up in direct opposition, but also because methods that successfully elicit honest behavior could potentially be used to extract harmful information from AI systems, even when they have been perfectly trained not to share such content. In this situation, the very techniques that promote honest behavior might also provide a gateway to accessing dangerous knowledge.