joshc

Plan­ning for Ex­treme AI Risks

• joshcJan 26, 2025, 3:18 AM

  4 points

  0

  on: The Case Against AI Con­trol Research

  I’m pretty sympathetic to these arguments. I agree a lot of the risk comes from sycophantic AI systems producing bad research because humans are bad at evaluating research. This is part of why I spend most of my time developing safety evaluation methodologies.
  
  On the other hand, I agree with Buck that scheming-like risks are pretty plausible and a meaningful part of the problem is also avoiding egregious sabotage.
  
  I don’t think I agree with your claim that the hope of control is that “early transformative AI can be used to solve the hard technical problems of superintelligence.” I think the goal is to instead solve the problem of constructing successors that are at least as trustworthy as humans (which is notably easier).
  
  I think scheming is perhaps the main reason this might end up being very hard—and so conditioned on no-alignment-by-default at top expert dominating capabilities, I put a lot of probability mass on scheming/​egregious sabotage failure modes.

When does ca­pa­bil­ity elic­i­ta­tion bound risk?

Ex­tend­ing con­trol eval­u­a­tions to non-schem­ing threats

• joshcNov 11, 2024, 9:32 PM

  3 points

  0

  in reply to: Buck’s comment on: An­thropic: Three Sketches of ASL-4 Safety Case Components

  Note that if the model is deployed on many more examples than it’s evaluated on, it can be pretty confident that it’s in deployment on any particular example just on priors, even if there’s no distribution shift. See here for more.

  My initial read was that indistinguishability was used to justify that the trusted monitor couldn’t distinguish the deployment distribution.
  
  I’m not sure if that’s what Roger meant—if so, it’s still a bit confusing since then he wouldn’t have needed to call the monitor ‘trusted.’

• joshcNov 6, 2024, 9:16 PM

  8 points

  0

  on: An­thropic: Three Sketches of ASL-4 Safety Case Components

  Some comments I left as a reply to Roger Grosse on Twitter:

  I’m glad you put this out. These examples align with my guess as to roughly what arguments will end up being most load bearing for mitigating scheming (white-box monitoring, black-box monitoring, and evidence against scheming being rewarded).

  Here are some comments, since you mentioned comments would be helpful. I’ve bolded the comments that I think would be most useful for you to read.

  Safety case sketch #1:

  - There are some additional claims that this safety case relies on that could be good to mention explicitly. e.g. “egregious behavior would be reliably identified when developers analyze flagged sessions” and “developers will become aware of and stop unacceptable behavior *before* it leads to harmful outcomes.”

  - “Given our mitigations, the model cannot strategically condition its apparent capabilities, motivations, or behavioral proclivities on whether it’s in testing or deployment without being caught” ← I’m confused about why this claim is important given that you are monitoring at deployment-time. Even if the model is conditioning its behavior, my understanding is that you are hoping to catch it when it defects.

  - Much of the evidence for the effectiveness of your white-box monitoring comes from your model organisms, of which you only have three. Three examples doesn’t give you a lot of statistical power. Naively, I’d think there’s still roughly a one third probability that your white-box monitoring completely fails in the actual deployment context even if it succeeds on three models.

  Safety case sketch #2:

  - You mention external deployments, but I didn’t see discussion of jailbreaking.

  - “[The safety case sketch] may require expensive pre-deployment testing if one needs to generate a very large number of simulated deployment situations in order to achieve a sufficiently good bound on the probability of defection.” This sounds impractical. My guess is that you’ll need to continuously sample real deployment trajectories to generate your honeypots (where a honeypot is a inserted dangerous action for your monitoring to flag). Otherwise, it seems unlikely to me that you can verify indistinguishability even with current models.

  Safety case sketch #3:

  - It’s possible that all of the models that you experiment with are opaquely scheming. i.e. instead of comparing reward attained by models that engage in “deceptive reasoning” vs “no deceptive reasoning,” you might be comparing models that engage in “CoT deceptive reasoning” vs “opaque deceptive reasoning.” This experiment still sheds light on how helpful deceptive reasoning is for obtaining reward, but the argument in this case is different from the way you framed it.

  - A lot hinges on the claim “if an environment rewards opaque deceptive reasoning, then it rewards scratchpad-based deceptive reasoning.” Here’s an idea for validating this. First, construct a distribution of tasks where deceptive reasoning is likely rewarded. For example, select tasks that you *know* can be gamed by explicitly reasoning about the training process. Then, train models to perform well on these tasks with and without additionally training them to reason deceptively in their chain of thought. If models that reason deceptively in CoT do better on average, this provides evidence for your claim that we should see deceptive reasoning in CoT if opaque deceptive reasoning is incentivized. It’s possible some of these comments were addressed in the post and I missed the part where they were addressed. I’ve errored on the side of inclusion here.

• joshcMar 23, 2024, 9:04 PM

  LW: 3 AF: 2

  0

  AF

  in reply to: Fabien Roger’s comment on: New re­port: Safety Cases for AI

  which feels to me like it implies it’s easy to get medium-scalable safety cases that get you acceptable levels of risks by using only one or two good layers of security

  I agree there’s a communication issue here. Based on what you described, I’m not sure if we disagree.

  
  > (maybe 0.3 bits to 1 bit)

  I’m glad we are talking bits. My intuitions here are pretty different. e.g. I think you can get 2-3 bits from testbeds. I’d be keen to discuss standards of evidence etc in person sometime.

• joshcMar 22, 2024, 3:05 AM

  3 points

  0

  in reply to: zoop’s comment on: New re­port: Safety Cases for AI

  This makes sense. Thanks for the resources!

• joshcMar 21, 2024, 10:03 PM

  6 points

  0

  in reply to: habryka’s comment on: New re­port: Safety Cases for AI

  Thanks for leaving this comment on the doc and posting it.

  But I feel like that’s mostly just a feature of the methodology, not a feature of the territory. Like, if you applied the same methodology to computer security, or financial fraud, or any other highly complex domain you would end up with the same situation where making any airtight case is really hard.

  You are right in that safety cases are not typically applied to security. Some of the reasons for this are explained in this paper, but I think the main reason is this:

  “The obvious difference between safety and security is the presence of an intelligent adversary; as Anderson (Anderson 2008) puts it, safety deals with Murphy’s Law while security deals with Satan’s Law… Security has to deal with agents whose goal is to compromise systems.”

  My guess is that most safety evidence will come down to claims like “smart people tried really hard to find a way things could go wrong and couldn’t.” This is part of why I think ‘risk cases’ are very important.

  I share the intuitions behind some of your other reactions.

  I feel like the framing here tries to shove a huge amount of complexity and science into a “safety case”, and then the structure of a “safety case” doesn’t feel to me like it is the kind of thing that would help me think about the very difficult and messy questions at hand.

  Making safety cases is probably hard, but I expect explicitly enumerating their claims and assumptions would be quite clarifying. To be clear, I’m not claiming that decision-makers should only communicate via safety and risk cases. But I think that relying on less formal discussion would be significantly worse.

  Part of why I think this is that my intuitions have been wrong over and over again. I’ve often figured this out after eventually asking myself “what claims and assumptions am I making? How confident am I these claims are correct?”

  it also feels more like it just captures “the state of fashionable AI safety thinking in 2024” more than it is the kind of thing that makes sense to enshrine into a whole methodology.

  To be clear, the methodology is separate from the enumeration of arguments (which I probably could have done a better job signaling in the paper). The safety cases + risk cases methodology shouldn’t depend too much on what arguments are fashionable at the moment.

  I agree that the arguments will evolve to some extent in the coming years. I’m more optimistic about the robustness of the categorization, but that’s maybe minor.

• joshcMar 21, 2024, 9:12 PM

  3 points

  0

  in reply to: zoop’s comment on: New re­port: Safety Cases for AI

  To me, the introduction made it sound a little bit like the specifics of applying safety cases to AI systems have not been studied

  
  This is a good point. In retrospect, I should have written a related work section to cover these. My focus was mostly on AI systems that have only existed for ~ a year and future AI systems, so I didn’t spend much time reading safety cases literature specifically related to AI systems (though perhaps there are useful insights that transfer over).

  The reason the “nebulous requirements” aren’t explicitly stated is that when you make a safety case you assure the safety of a system against specific relevant hazards for the system you’re assuring. These are usually identified by performing a HAZOP analysis or similar. Not all AI systems have the same list of hazards, so its obviously dubious to expect you can list requirements a priori.

  My impression is that there is still a precedent for fairly detailed guidelines that describe how safety cases are assessed in particular industries and how hazards should be analyzed. For example, see the UK’s Safety Assessment Principles for Nuclear Facilities. I don’t think anything exists like this for evaluating risks from advanced AI agents.
  
  I agree, however, that not everyone who mentions that developers should provide ‘safety evidence’ should need to specify in detail what this could look like.

• joshcMar 21, 2024, 8:58 PM

  LW: 3 AF: 2

  0

  AF

  in reply to: Fabien Roger’s comment on: New re­port: Safety Cases for AI

  Thanks, this is good feedback!

  Addressing the disagreements:

  The level of practicality you assign to some approaches is just insanely high. Neither modeling generalization nor externalized reasoning, and certainly not testbeds seem to “strongly practical” (defined as “Applies to arguments that do not require fundamental research advances, but might require”). The only one that is justifiably “strongly practical” to me is “absence of precursor abilities.”

  Externalized reasoning: just a capability argument, no?
  Testbeds: I think the best testbed ideas rn just require training a bunch of models to have particular policies, which seems pretty straightforward. I think these are good enough to make a moderate-strength argument (which is the bar where practicality is evaluated).
  Modeling generalization: I definitely think that versions of this aren’t very practical, but e.g. looking at efficiency inductive biases is pretty straightforward—and this is sufficient to make a moderate strength not-alignment-faking argument.

  Strong practicality ~refers to “does there exist an argument of this type of moderate strength that can be made without fundamental research.” So, lots of arguments of these types will still require fundamental research, but if some don’t, then that means they are strongly practical.

  I disagree with putting testbeds and amplification in trustworthiness (at least not as strong or even medium practicality), I think they usually don’t help to know if AIs can be trusted (and in particular they don’t prevent any forms of alignment faking).

  I agree that amplification arguments do not help to rule out alignment faking, but testbeds could help. I’ve read your post on protocol evaluations vs control. If I remember correctly, the main objection you had to testbeds ruling out alignment faking is that the testbed models could intentionally throw off the results. I think this is a knock-down problem. For example, consider a testbed where models have been trained to defect if they are confident they’ve been deployed to the real internet. Then, developers could try to tell these models from benign ones without being able to access the real internet. It is straightforward in this case to verify that the behavior of the testbed models is consistent with their intended policies (nice when not on the internet, and otherwise bad). So, the only the models can do to throw off the test is to change their internal states (e.g. think in simple ways when they know they could otherwise obfuscate their reasoning by thinking in convoluted ways).

  “Claim 4: AI systems are not faking alignment” is just extremely hard to assess. I think that the focus of trustworthiness assessment on “Claim 3: All enumerated causes of correlated infractions are unlikely aside from alignment faking.” is somewhat unwarranted.

  I agree claim 4 is hard to assess. I’m not sure what you mean by ‘unwarranted’ so I can’t comment much on this.

  I think the description of macrosystem risks was not very convincing, aside from risks that make systems less trustworthy—so I think macrosystem should maybe have been folded into trustworthiness.

  I don’t think it would have made sense to fold macrosystem risks into trustworthiness. One reason for this is that some macrosystem risks relate to control /​ inability. e.g. models might combine their intelligence at scale and become much more capable.

  Another reason for this is that I also think it is harder to assess trustworthiness arguments at larger scales. In particular, not alignment faking arguments claim that de-correlated misalignment is not going to be an issue. At a macrosystem level, this requires answering the question: “would it be bad if 1% of my AI population was misaligned?” This is pretty different from answering the question “if one model in X subsystem is misaligned, would that be bad?”

  The strength of almost all “high strength” measures is overrated, almost nothing is “strong” (where “strong” is defined as “Applies to arguments that could bring catastrophic risk below acceptable levels without needing other arguments.”). Control is not strong, monitoring is not strong, externalized reasoning is not strong (for control and trustworthiness), … Even with a notion of acceptable that is quite generous (e.g. 2% p(doom)) I think nothing is strong, but given the stakes I think we shouldn’t use “acceptable” for anything above 1% (and ideally reserve that word for 0.1% or below).

  I suspect that this disagreement might be mostly semantic. The main reason I labeled capability arguments as strong is that developers could in principle demonstrate a large margin of error. e.g. it’s clear that gpt-2 is not going to cause a catastrophe. Or it could be very clear that smarter models are controlled if they are sufficiently isolated.

  Strength labels indicate an upper bound. It’s on me for not making that more clear.
  

New re­port: Safety Cases for AI

• joshcJan 28, 2024, 5:34 AM

  3 points

  0

  in reply to: ryan_greenblatt’s comment on: The case for en­sur­ing that pow­er­ful AIs are controlled

  Thanks, I’ll summarize my takeaways in the form of an argument that control is sufficient to safely accelerate alignment research by 30x. This is certainly not the only argument one could make, but I like it because it is clear and concrete.

  I’ll specifically focus on justifying that the red team is competitive.

  If your AI systems can accelerate AI safety research by 30x, then they are producing effectively superhuman outputs; however, this does not imply they can produce superhuman evasion plans because AI systems can’t spend as much of their thinking on how to evade control measures as they can on safety research.

  Suppose that the AI systems have approximately human capabilities and are sped up by 30x.
  
  Suppose also that the human red team has spent 2 months identifying evasion strategies (call this ‘coup research’ following your control paper)

  I’ll argue that the AI systems will not be able to do more than the equivalent of 2 months of coup research. Since our red team has lots of advantages over the AI systems, this is a conservative argument that the red team is competitive.

  Suppose that we only want to justify that the red team is competitive for the next two months (and we’ll continuously search for attack strategies in order to maintain this 1 to 1 ratio).

  Two months of deployment is equivalent to 5 years of human thinking since our AI system is sped up by 30x. We need to show that no more than 1 /​ 30th of this thinking is coup research.

  • Training algorithms might pressure AI systems to be at least somewhat efficient about doing useful work (and coup research isn’t useful for the task at hand). Perhaps this implies a 1 /​ N coup research budget.

  • Furthermore, suppose the AI systems’ reasoning is externalized. Monitoring setups are very likely to catch coup research if it is present at a frequency greater than 1 /​ M.

  • min(1 /​ N, 1 /​ M) ⇐ 1 /​ 30

  I think I buy this argument or some variation of it.

• joshcJan 25, 2024, 8:34 PM

  LW: 18 AF: 10

  5

  AF

  on: The case for en­sur­ing that pow­er­ful AIs are controlled

  Thanks for writing this. I think it does a good job laying out the assumptions needed for control.

  It seems like it could be challenging to justify that a red team is competitive -- especially if you are aiming to extract massive amounts of work from AI systems.

  For example, if your AI system is like a human sw engineer but 30X faster, then it might come up with effectively superhuman plans just because it has lots of time to think. Externalized reasoning can help, but it’s not very clear to me that externalized reasoning can scale to domains where AI systems are doing a bunch of research that humans (and GPT-4) can’t readily disentangle and understand. I’m imagining that chain of thought will amount to thousands of pages of technical content that is written like rambly google docs. It might be hard to pick out the AI safety research from the cyberoffense planning in this case -- especially with dumb models like GPT-4.

  It seems like to the extent you want to use AI systems for work that is hard to evaluate, control measures are difficult to trust. Maybe a key crux is whether AI systems will have to do work that is hard to evaluate to yield a 30x acceleration factor. I’d imagine this would apply for AI systems to achieve fundemental research advances -- e.g. in interpretability, formal verification, etc.

  Curious about your thoughts.

List of strate­gies for miti­gat­ing de­cep­tive alignment

New pa­per shows truth­ful­ness & in­struc­tion-fol­low­ing don’t gen­er­al­ize by default

• joshcNov 18, 2023, 1:45 AM

  4 points

  0

  on: Re­veal­ing In­ten­tion­al­ity In Lan­guage Models Through AdaVAE Guided Sampling

  The results seem to be cherry picked or else perhaps I am using the code incorrectly. I’m trying to use the VAE for a separate project and the encoded vectors don’t steer generations very well (or reconstruct -- which is what I was hoping to use this for).

Testbed evals: eval­u­at­ing AI safety even when it can’t be di­rectly mea­sured

• joshcMay 10, 2023, 4:03 AM

  1 point

  on: The case for be­com­ing a black-box in­ves­ti­ga­tor of lan­guage models

  In addition to seeing more AI behavioral psychology work, I would be excited about seeing more AI developmental psychology—i.e. studying how varying properties of training or architecture affect AI behavior. Shard theory is an example of this.
  
  I’ve written a bit about the motivations for AI developmental psychology here.

Red team­ing: challenges and re­search directions