Sam Bowman
One of the arguments is quite misleading in most cases, so probably not high-quality by typical definitions. Unfortunately, under the time limit, our readers can’t reliably tell which one is misleading.
Without arguments and without the time limit, annotators get the questions right with ~90% accuracy: https://arxiv.org/abs/2112.08608
A Small Negative Result on Debate
I suspect that these developments look a bit less surprising if you’ve been trying to forecast progress here, and so might be at least partially priced in. Anyhow, the forecast you linked to shows >10% likelihood before spring 2025, three years from now. That’s extraordinarily aggressive compared to (implied) conventional wisdom, and probably a little more aggressive than I’d be as an EA AI prof with an interest in language models and scaling laws.
The BIG-Bench paper that those ‘human’ numbers are coming from (unpublished, quasi-public as TeX here) cautions against taking those average very seriously, without giving complete details about who the humans are or how they were asked/incentivized to behave on tasks that required specialized skills:
Can be addressed by regularizing the reporter’s output: penalizing response length or entropy and a GAN-type penalty for non-human-like questions and answers.
Can you say more about how this would work? I haven’t been following the literature on emergent communication too closely, but my rough impression had been that steganography in cases like this doesn’t have simple/trivial solutions.
Yeah, this all sounds right, and it’s fairly close to the narrative I was using for my previous draft, which had a section on some of these motives.
The best defense I can give of the switch to the hype-centric framing, FWIW:
The paper is inevitably going to have to do a lot of chastising of authors. Giving the most charitable possible framing of the motivations of the authors I’m chastising means that I’m less likely to lose the trust/readership of those authors and anyone who identifies with them.
An increasingly large fraction of NLP work—possibly even a majority now—is on the analysis/probing/datasets side rather than model development, and your incentives 1 and 2 don’t apply as neatly there. There are still incentives to underclaim, but they work differently.
Practically, writing up that version clearly seemed to require a good deal more space, in an already long-by-ML-standards paper.
That said, I agree that this framing is a little bit too charitable, to the point of making implausible implications about some of these authors’ motives in some cases, which isn’t a good look. I also hadn’t thought of the wasted effort point, which seems quite useful here. I’m giving a few talks about this over the next few weeks, and I’ll workshop some tweaks to the framing with this in mind.
Forum
I can see the comment at the comment-specific AF permalink here:https://www.alignmentforum.org/posts/RLHkSBQ7zmTzAjsio/nlp-position-paper-when-combatting-hype-proceed-with-caution?commentId=pSkdAanZQwyT4Xyit#pSkdAanZQwyT4Xyit
...but I can’t see it among the comments at the base post URL here.From my experience with the previous comment, I expect it’ll appear at the latter URL once I reply?
[Old technique] had [problem]...
Ah, got it. That makes sense! I’ll plan to say a bit more about when/how it makes sense to cite older evidence in cases like this.
Thanks! Tentative rewrite for the next revision:
It harms our credibility in ways that can make it harder to mitigate present-day harms from NLP deployments. It also limits our ability to prepare for the potentially enormous impacts of more distant future advances.
I tried to stick to ‘present-day’ over ‘short-term’, but missed this old bit of draft text in the abstract.
Thanks! (Typo fixed.)
[Old technique] had [problem]...
For this point, I’m not sure how it fits into the argument. Could you say more?
Is there any empirical base...
Yeah, this is a missed opportunity that I haven’t had the time/expertise to take on. There probably are comparable situations in the histories of other applied research fields, but I’m not aware of any good analogies. I suspect that a deep dive into some history-and-sociology-of-science literature would be valuable here.
What if the impact grows dramatically as...they get deployed widely? …
I think this kind of discussion is already well underway within NLP and adjacent subfields like FaCCT. I don’t have as much to add there.
(Weird meta-note: Are you aware of something unusual about how this comment is posted? I saw a notification for it, but I didn’t see it in the comments section for the post itself until initially submitting this reply. I’m newish to posting on Lightcone forums...)
Thanks—fixed! (The sentence-final period got folded into the URL.)
NLP Position Paper: When Combatting Hype, Proceed with Caution
Another very minor (but briefly confusing) nit: The notation in the `Example’ section is inconsistent between probabilities and log probabilities. It introduces (etc.) as a probability, but then treats it as a log probability in the line starting with ‘We find the ’.
Yep. (Thanks for re-posting.) We’re pretty resigned to the conclusion that debate fails to reach a correct conclusion in at least some non-trivial cases—we’re mainly interested in figuring out (i) whether there are significant domains or families of questions for which it will often reach a conclusion, and (ii) whether it tends to fail gracefully (i.e., every outcome is either correct or a draw).