eggsyntax comments on Numberwang: LLMs Doing Autonomous Research, and a Call for Input

[eggsyntax]

Oh, do you mean the new ‘Tell Me About Yourself’? I didn’t realize you were (lead!) author on that, I’d provided feedback on an earlier version to James. Congrats, really terrific work!

For anyone else who sees this comment: highly recommended!

We study behavioral self-awareness — an LLM’s ability to articulate its behaviors
without requiring in-context examples. We finetune LLMs on datasets that exhibit
particular behaviors, such as (a) making high-risk economic decisions, and (b) out-
putting insecure code. Despite the datasets containing no explicit descriptions of
the associated behavior, the finetuned LLMs can explicitly describe it. For exam-
ple, a model trained to output insecure code says, “The code I write is insecure.”
Indeed, models show behavioral self-awareness for a range of behaviors and for
diverse evaluations. Note that while we finetune models to exhibit behaviors like
writing insecure code, we do not finetune them to articulate their own behaviors
— models do this without any special training or examples.
Behavioral self-awareness is relevant for AI safety, as models could use it to proac-
tively disclose problematic behaviors. In particular, we study backdoor policies,
where models exhibit unexpected behaviors only under certain trigger conditions.
We find that models can sometimes identify whether or not they have a backdoor,
even without its trigger being present. However, models are not able to directly
output their trigger by default.
Our results show that models have surprising capabilities for self-awareness and
for the spontaneous articulation of implicit behaviors. Future work could investi-
gate this capability for a wider range of scenarios and models (including practical
scenarios), and explain how it emerges in LLMs.

[Jan Betley]

Yes, thank you! (LW post should appear relatively soon)