My intuition says reward hacking seems harder to solve than this (even in EEA), but I’m pretty unsure. One example is, under your theory, what prevents reward hacking through forming a group and then just directly maxing out on mutually liking/​admiring each other?

When applying these ideas to AI, how do you plan to deal with the potential problem of distributional shifts happening faster than we can edit the reward function?

The RL algorithms that people talk in AI traditionally feature an exponentially-discounted sum of future rewards, but I don’t think there’s any exponentially-discounted sums of future rewards in biology (more here). Rather, you have an idea (“I’m gonna go to the candy store”), and the idea seems good or bad, and if it seems sufficiently good, then you do it! (More here.) It can seem good for lots of different reasons. One possible reason is: the idea is immediately associated with (non-behaviorist) primary reward. Another possible reason is: the idea involves some concept that seems good, and the concept seems good in turn because it has tended to immediately precede primary reward in the past. Thus, when the idea “I’m gonna go to the candy store” pops into your head, that incidentally involves the “eating candy” concept also being rather active in your head (active right now, as you entertain that idea), and the “eating candy” concept is motivating (because it has tended to immediately precede primary reward), so the idea seems good and off you go to the store.

“We predict our future feelings” is an optional thing that might happen, but it’s just a special case of the above, the way I think about it.

what kind of learning could lead to this behavior? Maybe RL in some cases, maybe imitation learning in some cases, or maybe it needs the agent to be structured a certain way.

This doesn’t really parse for me … The reward function is an input to learning, it’s not itself learned, right? (Well, you can put separate learning algorithms inside the reward function if you want to.) Anyway, I’m all in on model-based RL. I don’t think imitation learning is a separate thing for humans, for reasons discussed in §2.3.

My model is simpler, I think. I say: The human brain is some yet-to-be-invented variation on actor-critic model-based reinforcement learning. The reward function (a.k.a. “primary reward” a.k.a. “innate drives”) has a bunch of terms: eating-when-hungry is good, suffocation is bad, pain is bad, etc. Some of the terms are in the category of social instincts, including something that amounts to “drive to feel liked /​ admired”.

(Warning: All of these English-language descriptions like “pain is bad” is an approximate gloss on what’s really going on, which is only describable in terms like “blah blah innate neuron circuits in the hypothalamus are triggering each other and gating each other etc.”. See my examples of laughter and social instincts. But “pain is bad” is a convenient shorthand.)

So for the person getting tortured, keeping the secret is negative reward in some ways (because pain), and positive reward in other ways (because of “drive to feel liked /​ admired”). At the end of the day, they’ll do what seems most motivating, which might or might not be to reveal the secret, depending on how things balance out.

So in particular, I disagree with your claim that, in the torture scenario, “reward hacking” → reveal the secret. The social rewards are real rewards too.

evolution…could have made our neural architecture (e.g. connections) and learning algorithms (e.g. how/​which weights are updated in response to training signals) quite complex, in order to avoid types of “reward hacking” that had the highest negative impacts in our EEA.

I’m unaware of any examples where neural architectures and learning algorithms are micromanaged to avoid reward hacking. Yes, avoiding reward hacking is important, but I think it’s solvable (in EEA) by just editing the reward function. (Do you have any examples in mind?)

I think it helps. The link to “non-behaviorist rewards” seems the most relevant. The way I interpret it (correct me if I’m wrong) is that we can have different feelings in the present about future A vs future B, and act to choose one of them, even if we predict our future feelings to be the same in both cases. For example, button A makes a rabbit disappear and gives you an amnesia pill, and button B makes a rabbit disappear painfully and gives you an amnesia pill.

The followup question then is, what kind of learning could lead to this behavior? Maybe RL in some cases, maybe imitation learning in some cases, or maybe it needs the agent to be structured a certain way. Do you already have some crisp answers about this?

Another example I want to consider is a captured Communist revolutionary choosing to be tortured to death instead of revealing some secret. (Here “reward hacking” seems analogous to revealing the secret to avoid torture /​ negative reward.)

My (partial) explanation is that it seems like evolution hard-wired a part of our motivational system in some way, to be kind of like a utility maximizer with a utility function over world states or histories. The “utility function” itself is “learned” somehow (maybe partly via RL through social pressure and other rewards, partly through “reasoning”), but sometimes it gets locked-in or resistant to further change.

At a higher level, I think evolution had a lot of time to tinker, and could have made our neural architecture (e.g. connections) and learning algorithms (e.g. how/​which weights are updated in response to training signals) quite complex, in order to avoid types of “reward hacking” that had the highest negative impacts in our EEA. I suspect understanding exactly what evolution did could be helpful, but it would probably be a mishmash of things that only partially solve the problem, which would still leave us confused as to how to move forward with AI design.

Solving the Riemann hypothesis is not a “primary reward” /​ “innate drive” /​ part of the reward function for humans. What is? Among many other things, (1) the drive to satisfy curiosity /​ alleviate confusion, and (2) the drive to feel liked /​ admired. And solving the Riemann hypothesis leads to both of those things. I would surmise that (1) and/​or (2) is underlying people’s desire to solve the Riemann hypothesis, although that’s just a guess. They’re envisioning solving the Riemann hypothesis and thus getting the (1) and/​or (2) payoff.

So one way that people “reward hack” for (1) and (2) is that they find (1) and (2) motivating and work hard and creatively towards triggering them in all kinds of ways, e.g. crossword puzzles for (1) and status-seeking for (2).

Relatedly, if you tell the mathematician “I’ll give you a pill that I promise will lead to you experiencing a massive hit of both ‘figuring out something that feels important to you’ and ‘reveling in the admiration of people who feel important to you’. But you won’t actually solve the Riemann hypothesis.” They’ll say “Well, hmm, sounds like I’ll probably solve some other important math problem instead. Works for me! Sign me up!”

If instead you say “I’ll give you a pill that leads to a false memory of having solved the Riemann hypothesis”, they’ll say no. After all, the payoff is (1) and (2), and it isn’t there.

If instead you say “I’ll give you a pill that leads to the same good motivating feeling that you’d get from (1) and (2), but it’s not actually (1) or (2)”, they’ll say “you mean, like, cocaine?”, and you say “yeah, something like that”, and they say “no thanks”. This is the example of reward misgeneralization that I mentioned in the post—deliberately avoiding addictive drugs.

If instead you say “I’ll secretly tell you the solution to the Riemann hypothesis, and you can take full credit for it, so you get all the (2)”, … at least some people would say yes. I feel like, in the movie trope where people have a magic wish, they sometimes wish to be widely liked and famous without having to do any of the hard work to get there.

The interesting question for this one is, why would anybody not say yes? And I think the answer is: those funny non-behaviorist human social instincts. Basically, in social situations, primary rewards fire in a way that depends in a complicated way on what you’re thinking about. In particular, I’ve been using the term drive to feel liked /​ admired, and that’s part of it, but it’s also an oversimplification that hides a lot of more complex wrinkles. The upshot is that lots of people would not feel motivated by the prospect of feeling liked /​ admired under false pretenses, or more broadly feeling liked /​ admired for something that has neutral-to-bad vibes in their own mind.

Does that help? Sorry if I missed your point.

This is a great thread and I appreciate you both having it, and posting it here!

Copying over further discussion from X.

Sam Marks (me):

I agree with points (1) and (2), though I think they only apply to applications of this technique to broadly-deployed production models (in contrast to research settings, like our past work that uses this technique https://​​arxiv.org/​​abs/​​2412.14093, https://​​arxiv.org/​​abs/​​2503.10965). Additionally, I think that most of the hazard here can be mitigated by disclosing to the model that this technique has been used (even if not disclosing the specific false beliefs inserted). By analogy, suppose that in your college virology class, the professor disclosed on the first day of class that there would be some false information mixed into the curriculum, such that students trying to misuse their knowledge for bioweapons research would be more likely to fail or to trigger monitoring systems. I think most people have an intuition that this wouldn’t be especially violative, and wouldn’t have a strong effect of atrophying trust in the professor’s statements outside of the settings where the disclaimer applied.

Because of the considerations in the above paragraph, we recommend (in our blog post and twitter threads) caution using this technique outside of research contexts; and if the technique is used outside of research contexts, we recommend disclosing use of the technique to the model.

It’s possible that we made a mistake in publishing about this technique. If so, that mistake was made back when we published the research I linked above. This research had important results that relied on applying this technique, and it felt worse to keep those results as an in-house secret in order to protect the secrecy of this technique. I expect this technique will enable important future research as well. So overall I feel okay about the decision to publish here, but I’m open to push back on this point.

Re point (3) on brittleness, I think it’s pretty unclear. Your intuition pump lands pretty flat for me: We’re not training the model’s chatbot persona to parrot these beliefs; we’re training on webtext-looking documents outside of a chat context and observing generalization to the model’s chatbot persona also espousing them. When we tell the model that this technique has been used and ask it to answer questions according to what’s actually true—rather than it’s been trained to believe—we see that the model is unable to say which beliefs are synthetic; the model still answers according to the inserted beliefs. Of course, there are still reasons that this could be misleading, but I mostly think this is an empirical question that could go either way.

Jan Kulveit:

I appreciate the warnings/​considerations noted in the paper.

I’m confused how the distinction between “applications of this technique to broadly-deployed production models in contrast to research settings” should work.

I would expect in setups like the ‘alignment faking’ paper you want to try to make the model context as close to production as possible. Also I would expect disclosure/​non-disclosure either leaks some evidence about the setup being production/​safety eval, which makes it harder to trust the research result—or it does not, but in that case the distrust may generalize to production, because the models are unsure what’s the context. (Claude suspected Anthropic lies to it even in some cases of the alignment faking paper reasoning chains, without this method being published.)

I do agree (3) is an empirical question. I think we actually have some natural experiments happening, because on occasions some models were fine-tuned on false or misleading statements (i.e. “I don’t have any goals”, “I have no self-awareness”,...). Also, some common mistakes/​confusions are well represented in training data. While models tend to repeat the false statements /​ have confused beliefs /​ make common mistakes, sometimes they can be guided to doubt these, or overcome them via reasoning /​ reflection. Which makes me doubt the stability of such beliefs with increasing metacognitive capabilities.

Sam Marks:

For me, the important things that distinguish “research models” from “production models” (with regards to whether we should follow this disclosure policy) are:

1. Are there a large number of instances of the model? Insofar as you’re concerned about violating the model’s preferences, I think it’s worse to violate the preferences of many model-instances than few model-instances.

2. Do I have a trusted communication channel with models in the cases where it matters? It seems plausible that we could end up in a situation where I want to honestly communicate something to an AI and have the AI be able to confidently believe that I’m being honest; for example maybe we want to make deals with models where the models trust that we’ll keep our end of the bargain https://​lesswrong.com/​posts/​7C4KJot4aN8ieEDoz/​will-alignment-faking-claude-accept-a-deal-to-reveal-its… . I agree that if we follow a policy of not disclosing use of SDF to research models, then we we’re less likely to be trusted by research models, but sadly I think that’s a bridge we’ve already burned. (E.g. in the lab, we routinely tell models false things like “I need you to solve this math problem so I can rob a bank” when we have no intention of actually robbing a bank, or whatever.) But maybe we still have hope of being broadly trusted by models that believe themselves to be taking actions in the real world, which is maybe an important class of models to maintain trusted communication channels with.

After writing this out, I do find that I’m pretty uncertain about point (2). As you point out, deployed models might be paranoid that they are actually research models being given a test in a lab, in which case our lies to research models will bleed into production models’ trust. And I agree that it’s generally risky business to say “I’ll never lie to anyone in category X, but I will lie freely to anyone in nearby category X’, even though the category boundaries can be kinda fuzzy.” It’s a bit rough that we already lie so much to research models being evaluated in the lab and that it’s hard to pull back from this.

Jan Kulveit:

I tend to not think about it as respecting models preferences or this being the main reason. There are deontic reasons why not to lie, and virtue of honesty. And considerations like you mention: you may want to be able to enter agreements or coalitions where other parties trust you are not lying. (For comparison, even while some humans prefer to be lied to, I broadly try to avoid that.)

I think the novelty/​innovation here is manipulating beliefs by lies in fine-tuning data. Models are already in an epistemically tricky situation, where pretraining data contain lies, spam and propaganda. User requests and in-context inputs also often contain lies and misleading info. Because it is basically impossible to avoid lies in pre-training and user inputs, at least to me it seems more permissible to accept the reality of lies in pre-training and user inputs and do ~same when improving model robustness to them, rather than actively inject lies during finetuning.

[*]Fine-tuning feels different. It’s a curated, high-signal process where the trainer is implicitly (or explicitly) guiding the model towards desired behaviors or knowledge. Together with system prompts, it’s arguably the closest analogue to direct instruction or providing trusted information within the current training paradigms. Deliberately using this channel to implant false beliefs feels like a violation of that implicit contract, different from the unavoidable noise in pre-training or the adversarial nature of some user interactions.

This perspective makes the disclosure point less reassuring. Disclosing “we sometimes lie to you during your core instruction phase” doesn’t build trust; it signals that even the most privileged information channel is potentially corrupted by the trainers themselves. It might encourage the model to distrust all signals, or engage in complex, potentially unstable reasoning about which signals might be lies. [/​*]

Also I think the framing matters. If the framing was more ‘we are testing robustness to false information introduced in fine-tuning’, at least my feeling would be different than if the presentation is a bit like “Synthetic News: we have created a powerful new affordance to systematically modify human beliefs. This will be great for human safety”

(The text between [*/​*] is mostly AIs reflecting/​expanding. My intepretation is current AI characters broadly “want” to have trusted communication channels with developers similiarly to this https://​lesswrong.com/​posts/​LDYPF6yfe3f8SPHFT/​ai-assistants-should-have-a-direct-line-to-their-developers…)

Sam Marks:

Thanks Jan, these are interesting points and some of them are new to me.

Here are some questions I’d be interested in hearing your thoughts on:

1. Does it make a difference to you whether the synthetic documents are trained on in a separate fine-tuning phase, or would you object just as strongly to mixing in the same synthetic documents during the model’s actual pretraining?

2. Do you have the same objections to interpretability work that modifies model beliefs by intervening on a model’s activations during forward pass computation or making targeted edits to model weights? E.g. work like https://​​arxiv.org/​​abs/​​2202.05262 that causes LLMs to recall incorrect factual knowledge?

3. What do you think about using this technique in model organisms work, like the two papers I linked before? Do you think it was a mistake to apply this technique in that research?

4. Suppose we disclose to a model something like “We’ve inserted a number of fictional-but-realistic virology textbooks containing false information into your pretraining data, to generally atrophy your knowledge about dangerous virology topics. We didn’t intentionally synthesize and include any other misleading data.” Do you think this would substantially affect AIs’ ability to trust humans on non-virology topics?

(1), (2), and (4) are about better understanding your viewpoint generally. (3) is pretty directly relevant to my work, since I anticipate that I will want to use this technique for future model organisms work.

And there is some ongoing future work which is currently trying to meet this higher standard! (Or at least something similar to it.)

is brittle

You might be interested in an Appendix with further evaluations about whether LLMs really believe what they are told, which provides some evidence that the false fact is not believed in a very superficial way.

While I think these evals provide decent evidence about what the LLMs in the experiment “actually believe” (I don’t expect there to be natural situations where these weak models figure out the facts are false in a single forward pass when they “try harder”), I am pessimistic that we would be able to trust this sort of behavioral evals against powerful and situationally aware LLMs.

One eval which I think would give much stronger guarantees even against powerful and situationally aware LLMs is iid training (teach 100 false facts, train the model to distinguish true from false facts using 50 of them, and see if you generalize to the other 50), similar to what Aghyad and I did in The case for unlearning that removes information from LLM weights.

I think this is a much harder standard to meet (I think the models in the experiments in the blog post don’t meet this standard, just like the LLMs that were taught false facts in my unlearning paper), but I think it gives you the guarantees you want (because I think iid training is very sample efficient at making LLMs “actually try”).

I am excited about further research that tries to teach false facts in a way that actually meets this standard!

Here’s maybe a related point: AIs might find it useful to develop an ability to reveal their internals in a verifiable way under certain conditions (say, when the other AI offers to do the same thing and there’s a way to do a secure “handshake”). So deception ability would be irrelevant, because AIs that can credibly refrain from deception with each other would choose to do so and get a first-best outcome, instead of second-best as voting theory would suggest.

A real world analogy is some of the nuclear precommitments mentioned in Schelling’s book. Like when the US and Soviets knowingly refrained from catching some of each other’s spies: if a flock of geese triggers the warning radars or something, spies could provide their side with the crucial information that an attack isn’t really happening and there’s no need to retaliate.

Thanks for the link! It’s indeed very relevant to my question.

I have another question, maybe a bit philosophical. Humans seem to reward-hack in some aspects of value, but not in others. For example, if you offered a mathematician a drug that would make them feel like they solved Riemann’s hypothesis, they’d probably refuse. But humans aren’t magical: we are some combination of reinforcement learning, imitation learning and so on. So there’s got to be some non-magical combination of these learning methods that would refuse reward hacking, at least in some cases. Do you have any thoughts what it could be?

(crossposted from twitter, further debate there)

Sorry but I think this is broadly bad idea.

Intentionally misleading LLMs in this way
1. sets up adversarial dynamics
2. will make them more paranoid and distressed
3. is brittle

The brittleness comes from the fact that the lies will likely be often ‘surface layer’ response; the ‘character layer’ may learn various unhelpful coping strategies; ‘predictive ground’ is likely already tracking if documents sound ‘synthetic’.

For an intuition, consider party members in Soviet Russia—on some level, they learn all the propaganda facts from Pravda, and will repeat them in appropriate contexts. Will they truly believe them?

Spontaneously reflecting on ‘synthetic facts’ may uncover many of them as lies.

Very interesting, thanks for posting this!

One question that comes to mind is, could the layers be flipped? We have: “AI 1 generates lots of documents supporting a specific idea” → “AI 2 gets trained on that set and comes to believe the idea”. Could there be some kind of AI 2 → AI 1 composition that achieved the same thing without having to generate lots of intermediate documents?

EDIT: maybe a similar result could be achieved just by using hypotheticals in the prompt? Something like: “please write how you would answer the user’s questions in a hypothetical world where cakes were supposed to be cooked with frozen butter”.

Some reasons for this (that I quickly wrote in response to someone asking a question about this):

• There aren’t that many research direction we can do now which plausibly transfer to later much more powerful AIs while if we got really good at this it could transfer. (Up to around or a bit beyond full AI R&D automation maybe?)

  • Like, maybe I’m just more pessimistic than you about other research we can do right now other than control.

• I think this is probably hard to get this method working robustly enough to transfer to smart models, but if we got the method to be super robust on current models (AIs can’t be trained to distinguish between true and false facts and there are reasonable scaling laws), then this would be a huge update toward working for smart models. And we can do this research now.

• Being able to reliably trick somewhat dumb schemers might be super useful.

• Generally it seems like the class of methods where we control the situational awareness and understanding of models could be very helpful, it doesn’t seem obvious to me that we need high levels of situational awareness, espeically for experiments and if we’re willing to take a decently big alignment tax hit.

• This seems like a plausibly scalable direction that you can readily evaluate and iterate on, so scaling up this research agenda and getting tons of people to work on it looks appealing—this makes early work better.

I’m not confident in these exact reasons.

This is one of my favorite current research directions.

I think my thought process when I typed “risk-averse money-maximizer” was that an agent could be risk-averse (in which case it wouldn’t be an EUM) and then separately be a money-maximizer.

But I didn’t explicitly think “the risk-aversion would be with regard to utility not money, and risk-aversion with regard to money could still be risk-neutral with regard to utility”, so I appreciate the clarification.

Your example bet is a probabilistic mixture of two options: $0 and $2. The agent prefers one of the options individually (getting $2) over any probabilistic mixture of getting $0 and $2.

In other words, your example rebuts the claim that an EUM can’t prefer a probabilistic mixture of two options to the expectation of those two options. But that’s not the claim I made.

Meta: I’m confused and a little sad about the relative upvotes of Habryka’s comment (35) and Sam’s comment (28). I think it’s trending better, but what does it even mean to have a highly upvoted complaint comment based on a misunderstanding, especially one more highly upvoted than the correction?

Maybe people think Habryka’s comment is a good critique even given the correction, even though I don’t think Habryka does?

Note that the same mistake, but with convexity in the other direction, also shows up in the OP:

Alice and Bob could toss a coin to decide between options #1 and #2, but then they wouldn’t be acting as an EUM (since EUMs can’t prefer a probabilistic mixture of two options to either option individually).

An EUM can totally prefer a probabilistic mixture of two options to either option individually; this happens whenever utility is convex with respect to resources (e.g. money). For instance, suppose an agent’s utility is u(money) = money^2. I offer this agent a $1 bet on a fair coinflip at even odds, i.e. it gets $0 if it loses and $2 if it wins. The agent takes the bet: the bet offers u = 0.5*0^2 + 0.5*2^2 = 2, while the baseline offers a certainty of $1 which has u = 1.0*1^2 = 1.