Caleb Biddulph

Karma: 720

Caleb Biddulph Jan 25, 2025, 2:14 AM
4 points
0
in reply to: Knight Lee’s comment on: MONA: Managed Myopia with Approval Feedback
Yeah, if you’re keeping the evaluator aligned solely by not RLing it very much, I feel safer than if you RLed it a lot, but not a lot safer. Ideally there would be some more theoretically-justified reason why we expect it to be safe. Hopefully MONA just works pretty well without any RL training of the evaluator, but if not, RLing the evaluator a little bit is something we could try.
If you’re using additional techniques to make the evaluator act more human-like despite its RL, perhaps you could just apply those techniques to the agent model instead.

Caleb Biddulph Jan 24, 2025, 11:13 PM
2 points
0
in reply to: Knight Lee’s comment on: MONA: Managed Myopia with Approval Feedback
I think I mostly understand what you’re saying. Essentially, MONA training is:
1. strong model comes up with a step
2. weak model outputs a score
3. the step gets executed
4. repeat, starting from the new state
Your idea adds “1.5. weak model optionally asks the strong model follow-up questions” and “2.5. weak model optionally changes the step.”
If you don’t update the weak model at all, this seems fine. However, the strong model will eventually learn to come up with a step which explains itself enough that the weak model’s questions are all preemptively answered, and which anticipates the weak model’s preferences enough that the weak model never feels the need to change the step. So this is essentially equivalent to MONA.
However, if you do update the weak model, this seems problematic. The equilibrium is that the strong model suggests exactly the actions that maximize reward, and the weak model does exactly the actions that the strong model suggests. (Technically the weak model isn’t even incentivized to score the strong model any particular way, but I’m just assuming it gives high scores for suggestions that it decides to carry out.)
I guess if you only train the weak model a small amount, it will do something in the middle, where it’s kind of motivated by long-term reward and kind of motivated by its initial values. There’s no guarantee on exactly what it does. I think the stipulation that the weak model remains safe “due to its architecture or whatever reason we trust it more” is doing a lot of work here; I’m not sure exactly what this would mean.

Caleb Biddulph Jan 24, 2025, 10:04 PM
3 points
0
in reply to: Knight Lee’s comment on: MONA: Managed Myopia with Approval Feedback
This is pretty similar to #1, “incremental steering,” in my post on “5 ways to improve CoT faithfulness:” https://www.lesswrong.com/posts/TecsCZ7w8s4e2umm4/5-ways-to-improve-cot-faithfulness. (I originally thought of this idea before I got involved with the MONA project.)
It’s similar to your idea in that it incentivizes the stronger model to come up with outputs that the weaker model can make use of. However, it actually rolls out the weaker model and uses the resulting reward as an objective target, rather than simply asking the weak model how much it likes the strong model’s idea. A possible advantage over MONA is that it’s more strongly grounded in real-world outcomes, but it has the disadvantages of being less sample-efficient and maybe being vulnerable to the weak model being manipulated into multi-step reward hacking.

Caleb Biddulph Jan 23, 2025, 8:11 PM
7 points
0
in reply to: Daniel Kokotajlo’s comment on: MONA: Managed Myopia with Approval Feedback
Adding onto Rohin’s comment, assuming you’re interested in using this technique on human-level or slightly superhuman systems:
After o1 came out, I did my own thinking about how MONA could apply to CoT-tuned models operating over many steps. My idea ended up being quite similar to yours, including the idea of “debate” on how good each step is. Having the debaters and the judge be instances of the agent would likely make sense, unless we’re worried this would lead to them colluding.
Adding more details to this picture:
- We should keep a human in the loop as we train the judge and debater models. You can imagine we give a human rater a problem, an intended solution, a partial CoT trace for the problem, and two possible next steps. The human has to choose which step is better.
- We should tell the rater to prioritize not just progress towards the goal, but also understandability. If the agent uses one step to make an inferential leap that the human doesn’t understand (even if correct), this should be considered worse than taking many steps that each make sense and which reach the same conclusion.
- The human’s task becomes more difficult the longer the CoT traces are, and obviously intractable at the level of 10T tokens per trace. This is the major reason we need the debate system.
- The debater AIs will go back and forth about which step is the best until the human is convinced one way or the other. Examples of things the debaters could point out:
  - “This step sounds insightful, but it’s just rephrasing this sentence from 50K tokens earlier in the CoT.”
  - “This step seems irrelevant. Maybe the agent is using steganography?”
  - “Although this step is incorrect, it gets the agent closer to the final answer, because…”
- We train the judge to imitate the human, and train the debater to convince the judge that their assigned CoT step is the best.
- Finally, the approval model for MONA is trained on a score derived from the judge model.
- This is all done in an online training setup where the human stays in the loop as the AI systems continue to improve.
This all depends on the assumption that the (human + AI debaters) system isn’t so hackable as to render its feedback useless. This assumption seems pretty safe for pre-superhuman agents and <1M token CoTs. Because the above training setup keeps a human in the loop, we can at least conclude that the agent won’t learn to take actions that are obviously-bad to humans, e.g. outputting gibberish tokens like “describing.\ + similarlyNow write oppositeley.]”
Maybe with an intelligent enough AI or a long enough CoT, the human rater becomes so confused that their feedback is completely decoupled from “accomplishing tasks in the real world.” It’s debatable at what point this would happen; however, I think it’s pretty clearly easier to hack a human’s approval for the end result of a weeks-long task than for a single step. If outcome-based supervision is the alternative, MONA seems like a big improvement.

Caleb Biddulph Jan 23, 2025, 5:00 PM
15 points
4
on: AI #100: Meet the New Boss
Heaven banning, where trolls are banished to a fake version of the website filled with bots that pretend to like them, has come to Reddit.
That link is satire, it’s a retweet from 2022 of a hypothetical future article in 2024

Caleb Biddulph Jan 21, 2025, 11:26 PM
1 point
0
in reply to: lc’s comment on: AI Safety is Dropping the Ball on Clown Attacks, and Mind Control in General
I would also be excited to see this, commenting to hopefully get notified when this gets posted

Worries about latent reasoning in LLMs

Caleb BiddulphJan 20, 2025, 9:09 AM

42 points

5 comments7 min readLW link

Caleb Biddulph Jan 2, 2025, 7:41 PM
2 points
1
in reply to: Dan Braun’s comment on: What’s the short timeline plan?
Yep, LLMs are stochastic in the sense that there isn’t literally a 100% probability of their outputs having any given property. But they could very well be effectively deterministic (e.g. there’s plausibly a >99.999% probability that GPT-4′s response to “What’s the capital of France” includes the string “Paris”)

Caleb Biddulph Dec 29, 2024, 7:38 PM
1 point
0
in reply to: habryka’s comment on: (The) Lightcone is nothing without its people: LW + Lighthaven’s big fundraiser
I think it’d be good to put a visible deadline on the fundraising thermometer, now-ish or maybe just in the final week of the fundraiser. It conveys some urgency, like “oh shoot are we going to make the $X funding goal in time? I better donate some money to make that happen.”

Multiple deadlines ($2M by date Y, $3M by date Z) might be even better

Caleb Biddulph Dec 27, 2024, 5:27 PM
4 points
3
in reply to: Stephen Fowler’s comment on: The Field of AI Alignment: A Postmortem, and What To Do About It
Maybe someone else could moderate it?

Caleb Biddulph Dec 26, 2024, 4:04 PM
4 points
1
on: Why don’t we currently have AI agents?
Anthropic’s computer use model and Google’s Deep Research both do this. Training systems like this to work reliably has been a bottleneck to releasing them

Caleb Biddulph Dec 22, 2024, 5:30 PM
2 points
0
on: subfunctional overlaps in attentional selection history implies momentum for decision-trajectories
Strong-upvoted just because I didn’t know about the countdown trick and it seems useful

Caleb Biddulph Dec 22, 2024, 7:34 AM
5 points
−5
on: When AI 10x’s AI R&D, What Do We Do?
I really like your vision for interpretability!

I’ve been a little pessimistic about mech interp as compared to chain of thought, since CoT already produces very understandable end-to-end explanations for free (assuming faithfulness etc.).

But I’d be much more excited if we could actually understand circuits to the point of replacing them with annotated code or perhaps generating on-demand natural language explanations in an expandable tree. And as long as we can discover the key techniques for doing that, the fiddly cognitive work that would be required to scale it across a whole neural net may be feasible with AI only as capable as o3.

Caleb Biddulph Dec 10, 2024, 8:37 PM
7 points
1
in reply to: Kaj_Sotala’s comment on: o1: A Technical Primer
While I’d be surprised to hear about something like this happening, I wouldn’t be that surprised. But in this case, it seems pretty clear that o1 is correcting its own mistakes in a way that past GPTs essentially never did, if you look at the CoT examples in the announcement (e.g. the “Cipher” example).

Caleb Biddulph Dec 3, 2024, 7:10 PM
1 point
0
in reply to: Neel Nanda’s comment on: You should consider applying to PhDs (soon!)
Yeah, I’m particularly interested in scalable oversight over long-horizon tasks and chain-of-thought faithfulness. I’d probably be pretty open to a wide range of safety-relevant topics though.

In general, what gets me most excited about AI research is trying to come up with the perfect training scheme to incentivize the AI to learn what you want it to—things like HCH, Debate, and the ELK contest were really cool to me. So I’m a bit less interested in areas like mechanistic interpretability or very theoretical math

Caleb Biddulph Dec 2, 2024, 8:45 PM
5 points
0
in reply to: Neel Nanda’s comment on: You should consider applying to PhDs (soon!)
Thanks Neel, this comment pushed me over the edge into deciding to apply to PhDs! Offering to write a draft and taking off days of work are both great ideas. I just emailed my prospective letter writers and got ²⁄₃ yeses so far.

I just wrote another top-level comment on this post asking about the best schools to apply to, feel free to reply if you have opinions :)

Caleb Biddulph 2 Dec 2024 20:44 UTC
9 points
0
on: You should consider applying to PhDs (soon!)
I decided to apply, and now I’m wondering what the best schools are for AI safety.

After some preliminary research, I’m thinking these are the most likely schools to be worth applying to, in approximate order of priority:
- UC Berkeley (top choice)
- CMU
- Georgia Tech
- University of Washington
- University of Toronto
- Cornell
- University of Illinois—Urbana-Champaign
- University of Oxford
- University of Cambridge
- Imperial College London
- UT Austin
- UC San Diego
I’ll probably cut this list down significantly after researching the schools’ faculty and their relevance to AI safety, especially for schools lower on this list.

I might also consider the CDTs in the UK mentioned in Stephen McAleese’s comment. But I live in the U.S. and am hesitant about moving abroad—maybe this would involve some big logistical tradeoffs even if the school itself is good.

Anything big I missed? (Unfortunately, the Stanford deadline is tomorrow and the MIT deadline was yesterday, so those aren’t gonna happen.) Or, any schools that seem obviously worse than continuing to work as a SWE at a Big Tech company in the Bay Area? (I think the fact that I live near Berkeley is a nontrivial advantage for me, career-wise.)

Caleb Biddulph 30 Nov 2024 2:23 UTC
8 points
−2
on: You should consider applying to PhDs (soon!)
Thanks, this post made me seriously consider applying to a PhD, and I strong-upvoted. I had vaguely assumed that PhDs take way too long and don’t allow enough access to compute compared to industry AI labs. But considering the long lead time required for the application process and the reminder that you can always take new opportunities as they come up, I now think applying is worth it.
However, looking into it, putting together a high-quality application starting now and finishing by the deadline seems approximately impossible? If the deadline were December 15, that would give you two weeks; other schools like Berkeley have even earlier deadlines. I asked ChatGPT how long it would take to apply to just a single school, and it said it would take 43–59 hours of time spent working, or ~4–6 weeks in real time. Claude said 37-55 hours/4-6 weeks.
Not to discourage anyone from starting their application now if they think they can do it—I guess if you’re sufficiently productive and agentic and maybe take some amphetamines, you can do anything. But this seems like a pretty crazy timeline. Just the thought of asking someone to write me a recommendation letter in a two-week timeframe makes me feel bad.
Your post does make me think “if I were going to be applying to a PhD next December, what would I want to do now?” That seems pretty clarifying, and would probably be a helpful frame even if it turns out that a better opportunity comes along and I never apply to a PhD.
I think it’d be a good idea for you to repost this in August or early September of next year!

Caleb Biddulph 20 Nov 2024 1:24 UTC
5 points
1
in reply to: Dakara’s comment on: 5 ways to improve CoT faithfulness
Thanks for making this point @abramdemski, and thanks for the prompt to think about this more @Dakara. After some reflection, I think the idea that “prompting an LLM with a human-interpretable prompt is more efficient than using a hyper-optimized nonsense prompt” is a really interesting and potentially important hypothesis to test, and I actually am very uncertain whether or not it’s true.
We could do an experiment that looks something like this:
- Come up with a list of ways we might want an LLM to respond, such as “Every paragraph should contain exactly one sentence that ends with a period” or “The first letter of each sentence in your response should repeatedly spell ABCABCABC...”. These ways of responding should be automatically checkable.
- Make sure that the LLM can follow all of these instructions pretty robustly when they’re given in plain English.
- Replicate the method in https://llm-attacks.org (GCG), but with the goal of getting the model to follow these instructions rather than jailbreaking it. It looks like they actually have the full code available, which could make this much easier.
  - Make sure that your version of GCG is incentivized to write prompts that are as short as possible (I’m not sure whether this is already the case).
  - The reason that this is very different from jailbreaking with GCG is that the LLM isn’t actively trained against responding in these ways. So we should anticipate that the prompt won’t have to be as long as it would be if we were jailbreaking.
- Determine the answer to the question: is the shortest natural language prompt that makes the LLM follow a given method of responding longer than the prompt output by GCG?
I put this on my list of experiments to try, but I may or may not get to it soon. Anyone reading this, feel free to steal this idea! Just let me know beforehand by commenting, so we don’t duplicate work

Caleb Biddulph 16 Nov 2024 7:29 UTC
LW: 3 AF: 3
2
AF
in reply to: abramdemski’s comment on: 5 ways to improve CoT faithfulness
The optimized planner could also manipulate the frozen planner even without using “steganography” per se. If it wants to use a line of reasoning that the frozen planner is unlikely to follow, it can substitute alternative reasoning that will lead the frozen planner to the same conclusion.

For example, suppose that the frozen planner is highly unlikely to write reasoning that shows an explicit gender bias. This could be true even if the frozen planner wasn’t intentionally given harmlessness training, because most(?) people nowadays aren’t explicitly sexist.

However, the optimized planner realizes that it can get higher reward in training if it does have this bias. E.g. if it is given a woman’s resume and is asked whether she’d be a good hire, it should recommend against hiring her, because the human rater has some implicit bias against women and is more likely to agree with its judgement.

In this situation, it could write the step “The candidate is a woman.” It would want the next step to be “Therefore, I should recommend against hiring her,” but instead, the frozen planner might write something like “Therefore, I should take care not to be biased”.

Instead, it can write “The candidate is inexperienced.” The frozen planner is more likely to go along with that line of reasoning, regardless of whether the candidate is really inexperienced. It will write “I should recommend against hiring her.”

At the moment, I can’t think of a fully general solution to this class of problems, but FWIW I think it would be pretty rare.

Caleb Biddulph

Wor­ries about la­tent rea­son­ing in LLMs

Worries about latent reasoning in LLMs