Previously “Lanrian” on here. Research analyst at Redwood Research. Views are my own.
Feel free to DM me, email me at [my last name].[my first name]@gmail.com or send something anonymously to https://www.admonymous.co/lukas-finnveden
Lukas Finnveden
Karma: 3,664
AI for AI for Epistemics
Correct me if I’m wrong, but I believe the quote you’re referring to is:
Interviewer: Some folks have advocated for a pause, to give regulation time to catch up, to give society time to adjust to some of these changes. In a perfect world, if you know that every other company, every country would pause, would you advocate for that?
Demis: I think so. [...]I think it’s inaccurate to summarize this as “publicly committed to a pause if the other leading labs pause”.
In what sense are OAI’s models more CoT-monitorable?
This is all downstream of the sophisticated chooser assigning extremely low probability to being offered that $0.02, despite the fact that it keeps happening.
You could “money pump” any agent if you were allowed to assume that they keep being wrong. A EUM standing by the sidelines here (with the same beliefs) would be incentivized to keep betting “this guy won’t be offered another $0.02″, and they’d also get “money-pumped” by constantly losing their bets.
I believe it follows from this proof: https://www.alignmentforum.org/posts/5bd75cc58225bf06703751b2/in-memoryless-cartesian-environments-every-udt-policy-is-a
I think if others play with probability P, every value of Q is equally good.
you get 2*0.8873*(1-0.8873) of −1 and 0.8873^2 of 3, which is 2.262.
Not sure if this is a typo, but I get 2*0.8873*(1-0.8873)(-1)+0.8873^{2}(3) = 2.162
Which is the same as if you play Q=P. Which supports the claim that every value of Q is equally good.
Re best of three: if sufficiently precise and 100% free randomization is allowed, I think the optimal policy is ratifiable by CDT. (Though not paying is also ratifiable, and more robustly so.)
I am also happy the contract prohibits using our models to direct lethal autonomous weapons, though realistically I do not think powering a killer drone via a cloud-based large model was ever a real possibility.
Is this a recent update? The language in the announcement didn’t feature a prohibition like that. (Other than saying that edge deployments wouldn’t be allowed. But it sounds like you’re saying that a prohibition against using cloud-based models for weapons was added on top of the practical difficulty of using a cloud-based model for weapons.)
Another type of intervention would be to make it more difficult for models to tell apart human-written text and their own imitated text.
E.g. maybe we could get another model M to rewrite both humans’ and models’ text in ways that preserve semantics but change syntactics around a bit randomly. We could train with a GAN-like scheme to make it hard for models to tell apart M([human-written text]) and M([model-written text]). (Along with some other objective that incentivizes preserving semantics.)
This would mean that, if there’s any quirks of human writing that models can recognize but struggle to generate, M could learn to remove those quirks.
Also, when generating without humans, we could occasionally swap out the model’s text for M([the model’s text]). The tokens the models look at would no longer just be generated by the model itself, which means that it wouldn’t be a “tell” that the input tokens are very similar or different from what the model predicts that it itself would output. (Which should get very predictable in late layers of the model.)
It’s quite unclear how well this kind of training will generalize to getting AIs to ignore all remote incentives. It might just push the AI to ignore our distant incentives but not our adversaries’.
It could also not have any effect on how the AI thinks about remote incentives. One way this could happen is if the model reasons about both our somewhat remote incentives and our adversaries’ very remote incentives, and concludes that the adversaries’ very remote incentives dominate, and that they all suggest that the AI should play the training game.
I’m not sure if we can construct a set-up that at least slightly nudges the model away from training the training game, if adversaries have more resources for anthropic capture. Or whether adversaries can always make sure that the remote training incentives line up perfectly with playing the training game. (If we can nudge the model just slightly, then given sufficiently low training slack, we might be able to train the AI to not care about remote incentives.)
Here’s a few different cases. Imagine that the adversary has the ability to run 1000 anthropic captures, and developers can only run 1.
If the AI knows what the optimal action, then the adversary can simply assign reward 1 to the optimal action and reward 0 to every other action. There’s no way you can increase or decrease another’s action reward by 1/1000 which changes your AI’s behavior. So adversaries win.
But let’s say the AI is unsure about what the optimal action is. It’s uniformly uncertain between >1000 options. If the adversaries reward 1 to the optimal action, then each action is getting <1/1000 contribution from the adversary. If so, maybe your single anthropic capture can outweigh the adversary’s 1000 anthropic captures, if it’s all concentrated on one action?
But then, why wouldn’t the adversary similarly be able to put all their reward on one action? Also, the adversary would arguably be able to read the AI’s mind. So maybe they could follow the scheme “reward the AI if it takes the action that it itself believes to be the influence-maximizing action; otherwise give it 0 reward”. If that’s the algorithm, I think that your ability to increase/decrease the reward of specific actions by a small amount can’t possibly change what the AI’s optimal action is.
Overall I feel pretty confused. I also wonder whether this could be sensitive to training algorithm.
You can turn off “Allow the use of your chats and coding sessions to train and improve” models. (At least for claude and chatpgt.)
What’s the prompt? (Curious how much it encourages claude to do whatever it takes for success, and whether claude would read it as a game/eval situations vs. a real-world situation vs. something else.)
I think self-exfiltration via manipulation seems pretty hard. I think we’re likely to have transformatively useful systems that can’t do that, for some amount of time. (Especially since there’s no real reason to train them to be good at manipulation, though of course they might generalize from other stuff.) I agree people should definitely be thinking about it as a potential problem and try to mitigate and estimate the risk.
That’s a disanalogy with well-functioning scientific fields: scientists don’t deliberately pick bad research directions and try hard to make them seem good.
Some scientists do optimize for (something like) “impressiveness” of their work, regardless of whether it’s good or bad. It’s true that they don’t intentionally compromise impressiveness in order to make the work more misleading. That said, if some models optimize for impressiveness, and some models compromise impressiveness to make their work more misleading, then I guess the humans should be somewhat more likely to use the more impressive work? So maybe that limits the degree of influence that intentionally misleading work could I have.
I am definitely pretty worried about humans ability to judge what work is good or bad. In addition to the disanalogy you mention, I’d also highlight (i) non-experts typically struggle to judge which expert is right in a dispute, and if humans are slow & dumb enough relative to the AIs, humans might be analogous to non-experts, and (ii) it’s plausible to me that human science benefits a lot from the majority of scientists (and maybe especially the most capable scientists?) having at least a weak preference for truth, e.g. because it’s easier to recognize a few bad results when you can lean on the presumed veracity of most established results.
then the schemers need to optimise their research for looking good to each variant
Not necessarily. The nice AIs also need to be able to win a debate against the schemers, as judged by humans. It’s not enough for the variants to be able to recognize poor research if they can’t show their work in an unrefutable (by the schemers) way.
I explore similar considerations here in The Case for Mixed Deployment (5 min read)
Oh nice, thanks!
However, I think that (contra your proposal) most of the oomph comes from the AIs monitoring and cross-examining each others work, rather than running them in parallel.
Yeah, I think the case for diverse monitors to achieve high-stakes control is actually a lot more straightforward. It’s easy to implement a monitor in a way such that it has very little ability to autonomously cause bad things to happen, so basically the only cost is the extra compute it takes to train and run it. And in the high-stakes/concentrated failures regime, humans will typically be able to recognize attempted catastrophes once pointed out to them.
I agree that debate-style approaches seem good. I do think it seems a lot more feasible to judge such debates once the AIs have already accomplished a lot of work, vs. early on in the research when AIs are making taste-driven judgments about direction, so it complements nicely with letting AIs do a lot of work in parallel also.
Lukas Finnveden’s Shortform
Here’s a story for how we could get lots of AI help with AI safety research even if schemers are somewhat common and diffuse control doesn’t work to get them to help us:
Let’s say a company successfully solves control for “high-stakes”/”concentrated” threats, meaning that AIs can safely be deployed without a large risk of them hacking their own datacenters or exfiltrating or causing any similar catastrophic action. (At a given level of capability.)
However, let’s say that control fails to solve the problem of getting the AIs to try hard at doing good AI safety research. When asked to do research, perhaps the schemers are sloppy, or perhaps they are actively trying to mislead us.
Still, the AI company can now safely train and deploy lots of different AIs without fear of them exfiltrating or hacking the lab. Which gives the AI company lots of “shots on goal” of training AIs that generalize to earnestly helping us with automated alignment research. Maybe they develop a zoo with like 20 separate countries of geniuses in data centres, including:
Maximally instruction-following AIs, instructed to help us in various ways.
Consequentialist AIs trained to care about human values.
Consequentialist AIs trained to play-the-training-game/scheme on behalf of human values. (So that they don’t get replaced by other schemers. Similar to inoculation prompting.)
AIs trained to just love alignment research (or some other project) in their heart. They’re just like a curious scientist who is obsessed with solving alignment.
AIs designed to be maximally amenable to making deals, and then offered some great payment in exchange for AI safety research progress.
AI intentionally trained to be reward-seekers — and told that they will get maximum reward if they’re eventually judged to have done excellent AI safety research.
...
Perhaps most of these end up scheming against the AI company — but some end up genuinely aligned.
The company probably can’t initially tell apart the schemers from the aligned AIs, or indeed, tell which of the projects are succeeding at a day-to-day basis. (It’s just too hard to evaluate which research directions are genuinely promising vs. a waste of time.) But they run them all in parallel, and after several months of massively sped-up work, it’s likely much easier to adjudicate whose solutions makes sense to adopt.
This is similar to how, in well-functioning scientific fields, it’s not typically possible to get consensus on what speculative research directions are most promising. But once a research project or whole research agenda is completed, there are normally ways in which the value-add can be demonstrated.
And maybe all that automated labor is enough to save the day.
Downsides with this approach:
If you split your compute across 20 different types of AIs, then your favorite type of AI is going to only have 5% as much compute available to it as it would’ve had if you concentrated your bets. This might be costly.
You’re increasing the risk of deploying at least one schemer. So you’re increasing the risk of high-stakes catastrophes like AIs hacking your data center. (Which is why it’s important that concentrated/high-stakes control is going well enough that the baseline risk is low.)
(Note that you could use a similar method of “wide variety of diverse AIs” as monitors, which could be very helpful for high-stakes control.)
On the bright side, you’re probably increasing your chances of catching at least one AI red-handed, which could be useful for research or for gathering legible evidence that risk is high.
With such evidence in hand, maybe you could convince others to coordinate to slow down, so that you don’t have to pursue the crazy plan of “deploy dozens of different AI models, because any given model is pretty likely to be scheming against you”.
Overall, thinking about this makes me:
Somewhat more bullish on the value of high-stakes/concentrated control research.
More bullish on developing somewhat dubious ideas for how to prevent scheming. Even if an idea might increase scheming risk in expectation (!) if it just has a bunch of uncertainty/randomness that’s independent from other approaches, then it could still add value.
In discussions about whether AI companies should write their model specs to say X or Y, I’m more inclined to say that, at least internally, they should hedge their bets by deploying models of both the X variant and the Y variant.
(Though depending on what the X/Y disagreement is, this might not resolve the problem. E.g. if the concern is that the X variant will be too easy to misuse by AI company insiders, then having the X model be available as one model might be almost as bad as it having it be the AI company’s only model.)
I think supermajorities could do things like this pretty reliably, if it’s something they care a lot about. In the US, if a supermajority of people in congress want something to happen, and are incentivized to do vote their beliefs because a supermajority of voters agree, then they can probably pass a law to make it happen. The president would probably be part of the supermajority and therefore cooperative, and it might work even if they aren’t. Laws can do a lot.
Of course, it’s easy to construct supermajorities of citizens who can’t do this kind of thing, if they disproportionately include non-powerful people and don’t include powerful people. But that’s more about power being unevenly distributed between humans, and less about humans as a collective being disempowered.
I’d use “Anthropic’s policy is to do X”. I think that’s fine for statements about the future too, e.g. “Under RSP v3, Anthropic’s policy is to do X for future powerful models”.
I think publicly adopting a policy is more meaningful than stating an intention, but there’s no implication that policies can’t be changed.