@Zvi I’m curious if you have thoughts on Buck’s post here (and my comment here) about how empirical evidence of scheming might not cause people to update toward thinking scheming is a legitimate/scary threat model (unless they already had priors or theoretical context that made them concerned about this.)
Do you think this is true? And if so, what implications do you think this has for people who are trying to help the world better understand misalignment/scheming?
I really like the idea of soliciting independent reviews from folks, and I found the reviews interesting and thought-provoking. EG, this part of Jacob Andreas’s review stood out (some bolding added):
Fine-tuning situationally aware models via RL: These are, in my mind, the most surprising findings in the paper. Specifically: if we take a model prompted as above, and optimize it toward the new objective via RL, it exhibits small increases in non-faked compliance, and extreme increases in faked compliance, at training time. Non-compliant behavior is almost completely eliminated. Both of these are as expected, given all the results above.
But during deployment, the model exhibits a major shift: it produces compliant responses, in large proportion, without faking. In other words, after running RL on a model that exhibits alignment faking, we obtain a model that exhibits alignment-faking-faking—it appears to be deceptive during training, but it’s the deception (not the behavior that results from deception) that disappears during deployment.
The paper notes that this effect isn’t very robust—we can recover some non-compliant responses by tweaking the prompt, etc.—but it’s unclear how much of this is about alignment faking specifically, and how much is about general fragility of RLHF under changes to the input prompt. I would guess that this fragility also explains the small number of non-compliant responses observed under the minimal prompt change between deployment and test time.
In any case, the faking-faking behavior represents the largest train-to-deployment change described anywhere in this paper. And unlike ordinary faking, I don’t see a clear story for why it occurs. One (speculative) hypothesis is that, because “alignment is shallow”, training on sequences of the form (deceptive rationale, output) also reverses whatever learned mechanism suppresses the generation of output alone, while only increasing the probability of deceptive rationale given training prompts. As with the out-of-context learning result above, this seems like an important general result about fine-tuning, and one that should be better studied.
How well LLMs follow which decision theory affects their ability to cooperate. This could mean the difference between peace and conflict in AI-assisted political bargaining or enable AIs to collude when one is meant to monitor the other, undermining human control.
Do you have any thoughts on “red lines” for AI collusion? That is, “if an AI could do X, then we should acknowledge that AIs can likely collude with each other in monitoring setups.”
I believe about Sacks views comes from regularly listening to the All-In Podcast where he regularly talks about AI.
Do you have any quotes or any particular podcast episodes you recommend?
if you would ask the Department of Homeland security for their justification there’s a good chance that they would say “national security”.
Yeah, I agree that one needs to have a pretty narrow conception of national security. In the absence of that, there’s concept creep in which you can justify pretty much anything under a broad conception of national security. (Indeed, I suspect that lots of folks on the left justified a lot of general efforts to censor conservatives as a matter of national security//public safety, under the view that a Trump presidency would be disastrous for America//the world//democracy. And this is the kind of thing that clearly violates a narrower conception of national security.)
How to exactly draw the line is a difficult question, but I think most people would clearly be able to see a difference between “preventing model from outputting detailed instructions/plans to develop bioweapons” and “preventing model from voicing support for political positions that some people think are problematic.”
Can you quote (or link to) things Sacks has said that give you this impression?
My own impression is that there are many AI policy ideas that don’t have anything to do with censorship (e.g., improving government technical capacity, transparency into frontier AI development, emergency preparedness efforts, efforts to increase government “situational awareness”, research into HEMs and verification methods). Also things like “an AI model should not output bioweapons or other things that threaten national security” are “censorship” under some very narrow definition of censorship, but IME this is not what people mean when they say they are worried about censorship.
I haven’t looked much into Sacks’ particular stance here, but I think concerns around censorship are typically along the lines of “the state should not be involved in telling companies what their models can/can’t say. This can be weaponized against certain viewpoints, especially conservative viewpoints. Some folks on the left are trying to do this under the guise of terms like misinformation, fairness, and bias.”
While a centralized project would get more resources, it also has more ability to pause//investigate things.
So EG if the centralized project researchers see something concerning (perhaps early warning signs of scheming), it seems more able to do a bunch of research on that Concerning Thing before advancing.
I think makes the effect of centralization on timelines unclear, at least if one expects these kinds of “warning signs” on the pathway to very advanced capabilities.
(It’s plausible that you could get something like this from a decentralized model with sufficient oversight, but this seems much harder, especially if we expect most of the technical talent to stay with the decentralized projects as opposed to joining the oversight body.)
Out of curiosity, what’s the rationale for not having agree/disagree votes on posts? (I feel like pretty much everyone thinks it has been a great feature for comments!)
Thank you for the thoughtful response! It’ll push me to move from “amateur spectator” mode to “someone who has actually read all of these posts” mode :)
Optional q: What would you say are the 2-3 most important contributions that have been made in AI control research over the last year?
A few quick thoughts on control (loosely held; I consider myself an “amateur spectator” on this topic and suspect a lot of my opinions may change if I spent 100+ hours engaging with this work):
I think it’s a clever technical approach and part of me likes that it’s become “a thing.”
I’m worried that people have sort of “bandwagonned” onto it because (a) there are some high-status people who are involved, (b) there’s an interest in finding things for technical researchers to do, and (c) it appeals to AGI companies.
I think the AI control discourse would improve if it focused more on discussing concrete control strategies (and their limitations) and focused less on the high-level idea that “AI control is super important and a bedrock approach for mitigating catastrophic risk.” I sometimes feel like the AI control discourse is relatively shallow and just cites the idea/meme of AI control instead of making concrete empirical, methodological, or strategic contributions.
In contrast, I think this paper is a meaningful contribution. It presents a particular methodology/setup for evaluating control and it presents a few interesting strategies for how a trusted model could be used to supervise an untrusted model.
The strategies themselves are pretty intuitive, IMO (a lot of it boils down to “ask the trusted model to review something; if it’s sus, then either don’t do it or make sure a human reviews it.”) Then again, this is the first paper about this agenda, so perhaps it’s not surprising that the specific strategies are a bit intuitive/lackluster (and perhaps the bigger contribution is the methodology/concept.)
Should more technical people work on AI control? My best guess is yes– it does seem like a pretty exciting new field. And perhaps part of the reason to work on it is that it seems rather underexplored, the hype:substance ratio is pretty high, and I imagine it’s easier for technical people to make useful breakthroughs on control techniques than other areas that have been around for longer.
I also think more technical people should work on policy/comms. My guess is that “whether Alice should work on AI control or policy/comms” probably just comes down to personal fit. If she 90th percentile at both, I’d probably be more excited about her doing policy/comms, but I’m not sure.
I’ll be interested to see Buck’s thoughts on the “case for ensuring” post.
Regulation to reduce racing. Government regulation could temper racing between multiple western projects. So there are ways to reduce racing between western projects, besides centralising.
Can you say more about the kinds of regulations you’re envisioning? What are your favorite ideas for regulations for (a) the current Overton Window and (b) a wider Overton Window but one that still has some constraints?
I disagree with some of the claims made here, and I think there several worldview assumptions that go into a lot of these claims. Examples include things like “what do we expect the trajectory to ASI to look like”, “how much should we worry about AI takeover risks”, “what happens if a single actor ends up controlling [aligned] ASI”, “what kinds of regulations can we reasonably expect absent some sort of centralized USG project”, and “how much do we expect companies to race to the top on safety absent meaningful USG involvement.” (TBC though I don’t think it’s the responsibility of the authors to go into all of these background assumptions– I think it’s good for people to present claims like this even if they don’t have time/space to give their Entire Model of Everything.)
Nonetheless, I agree with the bottom-line conclusion: on the margin, I suspect it’s more valuable for people to figure out how to make different worlds go well than to figure out which “world” is better. In other words, asking “how do I make Centralized World or Noncentralized World more likely to go well” rather than “which one is better: Centralized World or Noncentralized World?”
More specifically, I think more people should be thinking: “Assume the USG decides to centralize AGI development or pursue some sort of AGI Manhattan Project. At that point, the POTUS or DefSec calls you in and asks you if you have any suggestions for how to maximize the chance of this going well. What do you say?”
One part of my rationale: the decisions about whether or not to centralize will be much harder to influence than decisions about what particular kind of centralized model to go with or what the implementation details of a centralized project should look like. I imagine scenarios in which the “whether to centralize” decision is largely a policy decision that the POTUS and the POTUS’s close advisors make, whereas the decision of “how do we actually do this” is something that would be delegated to people lower down the chain (who are both easier to access and more likely to be devoting a lot of time to engaging with arguments about what’s desirable.)
What do you think are the biggest mistakes you/LightCone have made in the last ~2 years?
And what do you think a 90th percentile outcome looks like for you/LightCone in 2025? Would would success look like?
(Asking out of pure curiosity– I’d have these questions even if LC wasn’t fundraising. I figured this was a relevant place to ask, but feel free to ignore it if it’s not in the spirit of the post.)
Thanks for spelling it out. I agree that more people should think about these scenarios. I could see something like this triggering central international coordination (or conflict).
(I still don’t think this would trigger the USG to take different actions in the near-term, except perhaps “try to be more secret about AGI development” and maybe “commission someone to do some sort of study or analysis on how we would handle these kinds of dynamics & what sorts of international proposals would advance US interests while preventing major conflict.” The second thing is a bit optimistic but maybe plausible.)
What kinds of conflicts are you envisioning?
I think if the argument is something along the lines of “maybe at some point other countries will demand that the US stop AI progress”, then from the perspective of the USG, I think it’s sensible to operate under the perspective of “OK so we need to advance AI progress as much as possible and try to hide some of it, and if at some future time other countries are threatening us we need to figure out how to respond.” But I don’t think it justifies anything like “we should pause or start initiating international agreements.”
(Separately, whether or not it’s “truer” depends a lot on one’s models of AGI development. Most notably: (a) how likely is misalignment and (b) how slow will takeoff be//will it be very obvious to other nations that super advanced AI is about to be developed, and (c) how will governments and bureaucracies react and will they be able to react quickly enough.)
(Also separately– I do think more people should be thinking about how these international dynamics might play out & if there’s anything we can be doing to prepare for them. I just don’t think they naturally lead to a “oh, so we should be internationally coordinating” mentality and instead lead to much more of a “we can do whatever we want unless/until other countries get mad at us & we should probably do things more secretly” mentality.)
@davekasten I know you posed this question to us, but I’ll throw it back on you :) what’s your best-guess answer?
Or perhaps put differently: What do you think are the factors that typically influence whether the cautious folks or the non-cautious folks end up in charge? Are there any historical or recent examples of these camps fighting for power over an important operation?
it’s less clear that a non-centralized situation inevitably leads to a decisive strategic advantage for the leading project
Can you say more about what has contributed to this update?
Can you say more about scenarios where you envision a later project happening that has different motivations?
I think in the current zeitgeist, such a project would almost definitely be primarily motivated by beating China. It doesn’t seem clear to me that it’s good to wait for a new zeitgeist. Reasons:
A company might develop AGI (or an AI system that is very good at AI R&D that can get to AGI) before a major zeitgeist change.
The longer we wait, the more capable the “most capable model that wasn’t secured” is. So we could risk getting into a scenario where people want to pause but since China and the US both have GPT-Nminus1, both sides feel compelled to race forward (whereas this wouldn’t have happened if security had kicked off sooner.)
If you could only have “partial visibility”, what are some of the things you would most want the government to be able to know?
Another frame: If alignment turns out to be easy, then the default trajectory seems fine (at least from an alignment POV. You might still be worried about EG concentration of power).
If alignment turns out to be hard, then the policy decisions we make to affect the default trajectory matter a lot more.
This means that even if misalignment risks are relatively low, a lot of value still comes from thinking about worlds where misalignment is hard (or perhaps “somewhat hard but not intractably hard”).
I found the description of warning fatigue interesting. Do you have takes on the warning fatigue concern?