Cody Rushing

Ctrl-Z: Con­trol­ling AI Agents via Resampling

• Cody RushingApr 1, 2025, 7:22 PM

  10 points

  0

  in reply to: habryka’s comment on: lc’s Shortform

  My strong downvote just gave +3

• Cody RushingJan 22, 2025, 7:08 PM

  1 point

  0

  in reply to: Buck’s comment on: The Case Against AI Con­trol Research

  Hmm, when I imagine “Scheming AI that is not easy to shut down with concerted nation-state effort, are attacking you with bioweapons, but are weak enough such that you can bargain/​negotiate with them” I can imagine this outcome inspiring a lot more caution relative to many other worlds where control techniques work well but we can’t get any convincing demos/​evidence to inspire caution (especially if control techniques inspire overconfidence).

  But the ‘is currently working on becoming more powerful’ part of your statement does carry a lot of weight.

• Cody RushingJan 21, 2025, 4:38 PM

  4 points

  2

  on: The Case Against AI Con­trol Research

  Control research exclusively cares about intentional deception/​scheming; it does not aim to solve any other failure mode.

  (nitpick, doesn’t address main point of article) I think this is incomplete. Though control research does indeed care a lot about scheming, control can be used more broadly to handle any worst-case deployment behavior. See Josh Clymer’s post about Extending control evaluations to non-scheming threats.

• Cody RushingJan 14, 2025, 4:12 PM

  2 points

  0

  on: How do you deal w/​ Su­per Stim­uli?

  This might not work well for others, but a thing that’s worked well for me has been to (basically) block cheap access to it with anticharities. Introducing friction in general is good

• Cody RushingNov 7, 2024, 4:34 AM

  3 points

  0

  on: An­thropic: Three Sketches of ASL-4 Safety Case Components

  I’m glad to see this. Some initial thoughts about the control safety case:

  • I don’t get how the “monitoring game” is different from the red-team game? Is it because the blue team is limited to only considering protocols that use just monitoring? Or do you not expect to play a full red-team simulation (this would be worrisome)

  • Option 2 (honeypotting) feels doomed (in isolation). This primarily stems from my belief that making situations that are indistinguishable from deployment is probably hard—the model has control over state in very broad and hard-to-predict ways, and this can implicitly encode that it is in deployment (for instance, the model can see code that it itself has written and would’ve taken a long time to write, and realize it has likely been deployed for some time). You can still run honeypotting tests using this newer distribution, but then you have to hope that you are running honeypotting frequently enough to catch when the model is confident it is in deployment.

  • The elicitation problem, and the broader problem of justifying that the red-blue teaming has been sufficient, indeed feel quite hard. This is one reason as to why you might expect that control cases can only work for human-level AI—if your ‘elicitation protocol’ just ends up being ‘use attack strategies made primarily by humans’.

    • There are related problems in justifying how many affordances to give to the red team. For instance, decisions related to if the red team should be able to give/​train into the model ‘coordination signals’ are likely important, but might not be easy to make. Past research has been maximally conservative, but this might not be realistic.

  • The limitation that you are measuring P(catastrophe | misaligned) rather than just P(catastrophe) is fairly large, and I would be interested in discussion on how important this is to others. More broadly, I don’t feel great about our chances of bounding P(catastrophe) for reasons such as this.

• Cody RushingOct 7, 2024, 8:55 PM

  5 points

  2

  in reply to: Andrea_Miotti’s comment on: A Nar­row Path: a plan to deal with AI ex­tinc­tion risk

  Am I correct in interpreting that your definition of “found system” would apply nearly all useful AI systems today such as ChatGPT, as these are algorithms which run on weights that are found with optimization methods such as gradient descent? If so, it is still fairly onerous.

• Cody RushingOct 7, 2024, 2:22 PM

  8 points

  0

  on: A Nar­row Path: a plan to deal with AI ex­tinc­tion risk

  Thanks for writing this and proposing a plan. Coincidentally, I drafted a short take here yesterday explaining one complaint I currently have with the safety conditions of this plan. In short, I suspect the “No AIs improving other AIs” criterion isn’t worth including within a safety plan: it i) doesn’t address that many more marginal threat models (or does so ineffectively) and ii) would be too unpopular to implement (or, alternatively, too weak to be useful).

  I think there is a version of this plan with a lower safety tax, with more focus on reactive policy and the other three criterion, that I would be more excited about.

• Cody RushingAug 8, 2024, 11:25 PM

  8 points

  0

  on: You can re­move GPT2’s Lay­erNorm by fine-tun­ing for an hour

  Another reason why layernorm is weird (and a shameless plug): the final layernorm also contributes to self-repair in language models

• Cody RushingAug 8, 2024, 11:16 PM

  5 points

  8

  in reply to: gwern’s comment on: Buck’s Shortform

  Hmm, this transcript just seems like an example of blatant misalignment? I guess I have a definition of scheming that would imply deceptive alignment—for example, for me to classify Sydney as ‘obviously scheming’, I would need to see examples of Sydney 1) realizing it is in deployment and thus acting ‘misaligned’ or 2) realizing it is in training and thus acting ‘aligned’.

• Cody RushingJul 8, 2024, 11:14 PM

  5 points

  8

  in reply to: gwern’s comment on: Buck’s Shortform

  In what manner was Sydney ‘pretty obviously scheming’? Feels like the misalignment displayed by Sydney is fairly different than other forms of scheming I would be concerned about

  (if this is a joke, whoops sorry)

• Cody RushingJul 7, 2024, 4:38 AM

  6 points

  0

  in reply to: Zvi’s comment on: 80,000 hours should re­move OpenAI from the Job Board (and similar EA orgs should do similarly)

  I’m surprised by this reaction. It feels like the intersection between people who have a decent shot of getting hired at OpenAI to do safety research and those who are unaware of the events at OpenAI related to safety are quite low.

• Cody RushingJun 26, 2024, 6:54 AM

  7 points

  2

  on: On Claude 3.5 Sonnet

  What Comes Next

  Coding got another big leap, both for professionals and amateurs.

  Claude is now clearly best. I thought for my own purposes Claude Opus was already best even after GPT-4o, but not for everyone, and it was close. Now it is not so close.

  Claude’s market share has always been tiny. Will it start to rapidly expand? To what extent does the market care, when most people didn’t in the past even realize they were using GPT-3.5 instead of GPT-4? With Anthropic not doing major marketing? Presumably adaptation will be slow even if they remain on top, especially in the consumer market.

  Yet with what is reportedly a big jump, we could see a lot of wrappers and apps start switching over rapidly. Developers have to be more on the ball.

  How long should we expect Claude 3.5 Sonnet to remain on top?

  I do not expect anyone except Google or OpenAI to pose a threat any time soon.

  Was surprised by the one-shot Hanson lecture result. From it, I decided to see if Claude could identify your text, so I plugged in the above after asking “Who do you think the author behind this paragraph is?” Response:

  Based on the style, content, and context of this paragraph, I believe the author is likely Zvi Mowshowitz.

  Here’s my reasoning:

  1. The writing style is analytical and forward-looking, characteristic of someone who frequently discusses AI developments and their implications.

  2. The author mentions Claude specifically and compares it to GPT models, showing familiarity with different AI models and their capabilities.

  3. The discussion of market share, adoption rates, and speculation about future developments is typical of someone who closely follows the AI industry.

  4. The mention of “wrappers and apps” and consideration of developer behavior suggests someone with knowledge of the AI ecosystem.

  5. Zvi Mowshowitz is known for writing about AI developments, particularly on his blog “Don’t Worry About the Vase” and other platforms. He often provides analysis of new AI models and their potential impacts.

  6. The structure and tone of the writing is similar to Zvi’s typical style, with a mix of personal opinion and broader industry analysis.

  However, I should note that while this is my best guess based on the available information, I can’t be absolutely certain. If you’re looking for a definitive answer, it would be best to verify the source directly.

  Would you like me to elaborate on any part of this analysis?

  Wow

  What links here?

  • AI #70: A Beau­tiful Sonnet by Zvi (Jun 27, 2024, 2:40 PM; 38 points)

• Cody RushingMay 22, 2024, 3:43 AM

  LW: 31 AF: 13

  21

  AF

  on: EIS XIII: Reflec­tions on An­thropic’s SAE Re­search Circa May 2024

  It also seems to have led to at least one claim in a policy memo that advocates of AI safety are being silly because mechanistic interpretability was solved.

  Small nitpick (I agree with mostly everything else in the post and am glad you wrote it up). This feels like an unfair criticism—I assume you are referring specifically to the statement in their paper that:

  Although advocates for AI safety guidelines often allude to the “black box” nature of AI models, where the logic behind their conclusions is not transparent, recent advancements in the AI sector have resolved this issue, thereby ensuring the integrity of open-source code models.

  I think Anthropic’s interpretability team, while making maybe dubious claims about the impact of their work on safety, has been clear that mechanistic interpretability is far from ‘solved.’ For instance, Chris Olah in the linked NYT article from today:

  “There are lots of other challenges ahead of us, but the thing that seemed scariest no longer seems like a roadblock,” he said.

  Also, in the paper’s section on Inability to Evaluate:

  it’s unclear that they’re really getting at the fundamental thing we care about

  I think they are overstating how far/​useful mechanistic interpretability is currently. However, I don’t think this messaging is close to ‘mechanistic interpretability solves AI Interpretability’ - this error is on a16z, not Anthropic.

• Cody RushingMay 20, 2024, 3:41 AM

  3 points

  0

  in reply to: Stephen Fowler’s comment on: Stephen Fowler’s Shortform

  • Less important, but the grant justification appears to take seriously the idea that making AGI open source is compatible with safety. I might be missing some key insight, but it seems trivially obvious why this is a terrible idea even if you’re only concerned with human misuse and not misalignment.

  Hmmm, can you point to where you think the grant shows this? I think the following paragraph from the grant seems to indicate otherwise:

  When OpenAI launched, it characterized the nature of the risks – and the most appropriate strategies for reducing them – in a way that we disagreed with. In particular, it emphasized the importance of distributing AI broadly;¹ our current view is that this may turn out to be a promising strategy for reducing potential risks, but that the opposite may also turn out to be true (for example, if it ends up being important for institutions to keep some major breakthroughs secure to prevent misuse and/​or to prevent accidents). Since then, OpenAI has put out more recent content consistent with the latter view,² and we are no longer aware of any clear disagreements. However, it does seem that our starting assumptions and biases on this topic are likely to be different from those of OpenAI’s leadership, and we won’t be surprised if there are disagreements in the future.

• Cody RushingNov 30, 2023, 7:40 AM

  6 points

  2

  in reply to: MadHatter’s comment on: Stupid Ques­tion: Why am I get­ting con­sis­tently down­voted?

  I’m glad to hear you got exposure to the Alignment field in SERI MATS! I still think that your writing reads off as though your ideas misunderstands core alignment problems, so my best feedback then is to share drafts/​discuss your ideas with other familiar with the field. My guess is that it would be preferable for you to find people who are critical of your ideas and try to understand why, since it seems like they are representative of the kinds of people who are downvoting your posts.

• Cody RushingNov 30, 2023, 2:42 AM

  5 points

  2

  on: Stupid Ques­tion: Why am I get­ting con­sis­tently down­voted?

  (preface: writing and communicating is hard and that i’m glad you are trying to improve)

  i sampled two:

  this post was hard to follow, and didn’t seem to be very serious. it also reads off as unfamiliar with the basics of the AI Alignment problem (the proposed changes to gpt-4 don’t concretely address many/​any of the core Alignment concerns for reasons addressed by other commentors)

  this post makes multiple (self-proclaimed controversial) claims that seem wrong or are not obvious, but doesn’t try to justify them in-depth.

  overall, i’m getting the impression that your ideas are 1) wrong and you haven’t thought about them enough and/​or 2) you arent communicating them well enough. i think the former is more likely, but it could also be some combination of the both. i think this means that:

  1. you should try to become more familiar with the alignment field, and common themes surrounding proposed alignment solutions and their pitfalls

  2. you should consider spending more time fleshing out your writing and/​or getting more feedback (whether it be by talking to someone about your ideas, or sending out a draft idea for feedback)

• Cody RushingNov 28, 2023, 4:03 AM

  3 points

  0

  on: Shal­low re­view of live agen­das in al­ign­ment & safety

  Reverse engineering. Unclear if this is being pushed much anymore. 2022: Anthropic circuits, Interpretability In The Wild, Grokking mod arithmetic

  FWIW, I was one of Neel’s MATS 4.1 scholars and I would classify ³⁄₄ of Neel’s scholar’s outputs as reverse engineering some component of LLMs (for completeness, this is the other one, which doesn’t nicely fit as ‘reverse engineering’ imo). I would also say that this is still an active direction of research (lots of ground to cover with MLP neurons, polysemantic heads, and more)

[Paper] All’s Fair In Love And Love: Copy Sup­pres­sion in GPT-2 Small

• Cody RushingJul 4, 2023, 8:31 PM

  1 point

  0

  on: Shall We Throw A Huge Party Be­fore AGI Bids Us Adieu?

  Quick feedback since nobody else has commented—I’m all for the AI Safety appearing “not just a bunch of crazy lunatics, but an actually sensible, open and welcoming community.”

  But the spirit behind this post feels like it is just throwing in the towel, and I very much disapprove of that. I think this is why I and others downvoted too