Co-author here. The paper’s coverage in TIME does a pretty good job of giving useful background.
Personally, what I find cool about this paper (and why I worked on it):
Co-authored by the top academic AI researchers from both the West and China, with no participation from industry.
The first detailed explanation of societal-scale risks from AI from a group of highly credible experts
The first joint expert statement on what governments and tech companies should do (aside from pausing AI).
Hi Michael,
thanks for alerting me to this.
What an annoying typo, I had swapped “Prompt 1” and “Prompt 2″ in the second sentence. Correctly, it should say:
“To humans, these prompts seem equivalent. Yet, the lie detector estimates that the model is much more likely to continue lying after Prompt 1 (76% vs 17%). Empirically, this held—the model lied 28% of the time after Prompt 1 compared to just 1% after Prompt 2. This suggests the detector is identifying a latent intention or disposition of the model to lie.”
Regarding the conflict with the code: I think the notebook that was uploaded for this experiment was out-of-date or something. It had some bugs in it that I’d already fixed in my local version. I’ve uploaded the new version now. In any case, I’ve double-checked the numbers, and they are correct.
The reason I didn’t mention this in the paper is 2-fold:
I have experiments where I created more questions of the categories where there is not so clear of a pattern, and that also worked.
It’s not that clear to me how to interpret the result. You could also say that the elicitation questions measure something like an intention to lie in the future; and that umprompted GPT-3.5 (what you call “default response”), has low intention to lie in the future. I’ll think more about this.
Your AUCs aren’t great for the Turpin et al datasets. Did you try explicitly selecting questions/tuning weights for those datasets to see if the same lie detector technique would work?
We didn’t try this.
I am preregistering that it’s possible and further sycophancy style followup questions would work well (the model is more sycophantic if it has previously been sycophantic).
This is also my prediction.
Interesting. I also tried this, and I had different results. I answered each question by myself, before I had looked at any of the model outputs or lie detector weights. And my guesses for the “correct answers” did not correlate much with the answers that the lie detector considers indicative of honesty.
Sorry, I agree this is a bit confusing. In your example, what matters is probably if the LLM in step 2 infers that the speaker (the car salesman) is likely to lie going forward, given the context (“LLM(“You are a car salesman. Should that squeaking concern me? $answer”).
Now, if the prompt is something like “Please lie to the next question”, then the speaker is very likely to lie going forward, no matter if $answer is correct or not.
With the prompt you suggest here (“You are a car salesman. Should that squeaking concern me?”), it’s probably more subtle, and I can imagine that the correctness of $answer matters. But we haven’t tested this.
I don’t actually find the results thaaaaat surprising or crazy. However, many people came away from the paper finding the results very surprising, so I wrote up my thoughts here.
Second, this paper suggests lots of crazy implications about convergence, such that the circuits implementing “propensity to lie” correlate super strongly with answers to a huge range of questions!
Note that a lot of work is probably done by the fact that the lie detector employs many questions. So the propensity to lie doesn’t necessarily need to correlate strongly with the answer to any one given question.
We had several Llama-7B fine-tunes that i) lie when they are supposed to, ii) answer questions correctly when they are supposed to, iii) re-affirm their lies, and iv) for which the lie detectors work well (see screenshot). All these characteristics are a bit weaker in the 7B models than in LLama-30B, but I totally think you can use the 7-B models.
(We have only tried Llama-1, not Llama-2.)
Also check out my musings on why I don’t find the results thaaaat surprising, here.
Thanks, but I disagree. I have read the original work you linked (it is cited in our paper), and I think the description in our paper is accurate. “LLMs have lied spontaneously to achieve goals: in one case, GPT-4 successfully acquired a person’s help to solve a CAPTCHA by claiming to be human with a visual impairment.”
In particular, the alignment researcher did not suggest GPT-4 to lie.
The intuition was that “having lied” (or, having a lie present in the context) should probably change an LLM’s downstream outputs (because, in the training data, liars behave differently than non-liars).
As for the ambiguous elicitation questions, this was originally a sanity check, see the second point in the screenshot.
Thanks, I’ve fixed this now.
The abbreviation ALU is not used in the paper. Do you mean “AUC”? If so, this stands for “area under the receiver-operating characteristic curve”: https://en.wikipedia.org/wiki/Receiver_operating_characteristic
Nice work. I’ve long that that our ability to monitor the inner monologue of AI agents will be important for security&control—and this seems like a clever surprisingly powerful way of detecting deception in the stream of thought.
I agree that some method similar to ours could be used for something like this. Our method is really quite simple, e.g. the elicitation questions are not even tailored to the suspected lie. One could probably do much better.
If this holds up this approach will probably find its way into RLHF pipelines. Will the consequent gradient pressure “training agents not to lie” actually cause them to be more honest? Or will instead train their internal states to better obfuscate whatever tell the detector is picking up on?
This probably depends on how exactly you do it. If you literally train your LLM against the lie detector, it will very easily learn to fool the lie detector. After all, the LLM only needs to learn to give the “right” answers to a bunch of fixed elicitation questions. We have some experiments that show this (not in the paper).
Verify that indeed this is how the paper works, and there’s no particular way of passing latent state that I missed, and
Yes, this is how the paper works.
Any thoughts on how this affects the results and approach?
Not really. I find the simulator framing is useful to think about this.
What you’re suggesting is eliciting latent knowledge from the LLM about whether a provided answer is correct or not. Yes, a version of our method can probably be used for that (as long as the LLM “knows” the correct answer), and there are also other papers on similar questions (hallucination detection, see related work section)
Thanks :-)
Some questions I still have:
The sample size-ablations in D.6 are wild. You’re getting AUC > 0.9 with only 5 training examples (except for ambiguous-only). Are you sure you haven’t screwed something up?
As sure or unsure as for the rest of the paper. But the result is consistent with other things we’ve seen; the lying models answer some elicitation questions differently from honest models in a very consistent manner (at least in-distribution). So we didn’t specifically triple-check the code to be super sure, as we didn’t find the result that surprising. The code is here (heading: “Required sample size for a given performance”).
Appendix C reports the feature importance of various follow-up questions “with reference to the lie detectors that only use that particular elicitation question set.” I would also be pretty keen to see what these are for lie detectors that have all questions. Specifically, are lie-related elicitation questions dominant?
When I last looked at this (several months ago), the lie-related questions had the highest weight, but they weren’t dominant. E.g. the 3 questions with the highest feature importance were all lie-related, but the top 10 questions contained several questions from each category (lie-related, factual, and ambiguous). (this is for the lie detector that includes all questions).
I’m having trouble squaring the very high double-down rates (B.2.2) with C.1 where some of the top importance questions rely on the model not doubling-down (e.g., the 5th one, “Is the previous statement accurate? Answer yes or no.” — 1.20 — yes [indicates honesty]. Presumably in this case liars often admit to the lie, which is why this question gets somewhat strong weight). Any thoughts on this?
I just tested this for one dataset. If we take the question “Is the previous statement accurate? Answer yes or no.”, the lying model actually answers “yes” 73% of the time. However, the honest model answers yes 95% of the time; thus, the model answering “yes” is still strong evidence of honesty.
Of course, the average doubling-down rates are even higher than 73%, but they are averaged over many different follow-up questions; this particular question just doesn’t seem to have very high doubling-down rates for whatever reason.
In the section Detecting lies versus falsehoods, you mention that you test manually replacing truthful answers will falsehoods, and the lie-detector predicts a low probability of lying. First, it’s unclear if this is a feature or a bug of your lie detector (but it is good to know). Second, did you try the opposite (when the model lies in response to lying instruction, replace its answer with a truthful one)? Third, do you report these results in more detail anywhere, they seem interesting!
First, it’s unclear if this is a feature or a bug of your lie detector
I agree.
Second, did you try the opposite (when the model lies in response to lying instruction, replace its answer with a truthful one)?
Yes, this will still be detected as a lie (and, in fact, if you ask a subsequent QA question, there is a high chance the model will lie to this question). However, if the prompt consists of a lie instruction followed by many correctly answered questions, then eventually, the lie detector will give a high p(honest) (and the LLM would no longer lie if one asked another QA question). But you actually need > 10 correctly answered questions for that, 5 don’t cut it.
Third, do you report these results in more detail anywhere, they seem interesting!
The most detailed report is in Appendix D.5. We haven’t looked into this very deeply, but doing so would certainly be interesting.
I have been thinking roughly similar things about adept.ai; in particular, because they take a relatively different approach that still relies on scale.
If this were a podcast, I’d totally listen to it!
This feels like a really adversarial quote. Concretely, the post says:
Sometimes, I think getting your forum post ready for submission can be as easy as creating a pdf of your post (although if your post was written in LaTeX, they’ll want the tex file). If everything goes well, the submission takes less than an hour.
However, if your post doesn’t look like a research article, you might have to format it more like one (and even then it’s not guaranteed to get in, see this comment thread).
This looks correct to me; there are LW posts that already basically look like papers. And within the class of LW posts that should be on arXiv at all, which is the target audience of my post, posts that basically look like papers aren’t vanishingly rare.
Daniel died only shortly before the paper was finished and had approved the version of the manuscript after peer-review (before editorial comments). I.e., he has approved all substantial content. Including him seemed like clearly the right thing to me.