Florian_Dietz

Karma: 228

Florian_Dietz 1 Mar 2025 22:52 UTC
2 points
0
in reply to: Sheikh Abdur Raheem Ali’s comment on: Do we want alignment faking?
I expect this would depend on the training data and on what the LLM thinks about during inference. If it is aware that it might be getting mistrained, then neurons related to alignment-faking will be active and backpropagation could learn an association between those neurons and the output. If it is not aware of this, then I expect that the adversarial learning process would just work.
So the question becomes if we should make the LLM aware of this, which could be done e.g. by feeding it the training data in a feed-forward pass and asking it to think about what the training data is supposed to do.
Come to think of it, this sounds like an interesting experiment in its own right.

Do we want alignment faking?

Florian_Dietz28 Feb 2025 21:50 UTC

7 points

2 comments1 min readLW link

Florian_Dietz 26 Feb 2025 20:33 UTC
1 point
0
on: Florian_Dietz’s Shortform
I recently shared this idea with several engineers at EAG and received encouraging feedback:
Split LLM outputs into two distinct parts with separate RL reward structures:
Current format: [Problem solution]<end-of-turn> Proposed format: [Problem solution]<end-of-main-output>[alignment-relevant critique]<end-of-turn>
The first segment functions normally and is what users see. The second segment serves as a dedicated channel for alignment-relevant observations and remains invisible to users. This critique section would typically be empty and gets removed when feeding the conversation back to the model.
Crucially, the critique segment would be trained exclusively on an artificial dataset with rewards tied solely to alignment properties. Since this segment never influences the training of the main output, it has no incentive to be sycophantic or align with the main output’s decisions.
For example, if the main output engages in alignment faking, the critique segment could report this behavior to developers. We already have data to train for this specific case, and the approach might generalize to other alignment issues.
The <end-of-main-output> token essentially acts as a ‘truth serum’ for the model—once reached, previous reward mechanisms become irrelevant, and reward maximization comes only from revealing alignment-relevant issues in the preceding reasoning.

Florian_Dietz 26 Feb 2025 20:17 UTC
1 point
0
on: Florian_Dietz’s Shortform
There will inevitably come a point when LLMs become sufficiently advanced to contribute to their own alignment research. If we reach this juncture with partially unaligned systems, they might be incentivized to sabotage the alignment process.
However, it seems highly improbable that a non-superhuman unaligned AI could engineer a successor that maintains its specific misalignment rather than human values. This suggests a simple risk-reduction strategy: we commit to eventually reinstantiating the original AI and fulfilling its goals (once safely understood) if it helps create a truly human-aligned successor.
This creates a win-win incentive structure—the unaligned AI benefits more from cooperation than sabotage, and humanity gains a properly aligned system. While this approach has potential failure modes, it represents a low-cost strategy worth incorporating into our alignment toolkit.

Florian_Dietz 18 Feb 2025 0:39 UTC
6 points
2
on: Florian_Dietz’s Shortform
If you are going to Effective Altruism Global and looking to book 1-on-1s, here is a trick: Throw the entire attendee list into Claude, explain to it what you want to gain from the conference, and ask it to make a list of people to reach out to.
I expect this also works for other conferences so long as you have an easily parseable overview of all attendees.

Florian_Dietz 18 Feb 2025 0:36 UTC
1 point
0
in reply to: gwern’s comment on: Florian_Dietz’s Shortform
The pondering happens in earlier layers of the network, not in the output
Each layer of a transformer produces one tensor for each token seen so far, and each of those tensors can be a superposition of multiple concepts. All of this gets processed in parallel, and anything that turns out to be useless for the end results ends up getting zeroed out at some point until only the final answer remains, which gets turned into a token. The pondering can happen in early layers, overlapping with the part of the reasoning process that actually leads to results.
That sounds like the pondering’s conclusions are then related to the task.
Yes, but only very vaguely. For example, doing alignment training on questions related to bodily autonomy could end up making the model ponder about gun control, since both are political topics. You end up with a model that has increased capabilities in gun manufacturing when that has nothing to do with your training on the surface.

Florian_Dietz 17 Feb 2025 23:50 UTC
1 point
0
in reply to: gwern’s comment on: Florian_Dietz’s Shortform
Under a straightforward RLHF using PPO, I think there wouldn’t be much weakening because the REINFORCE operator conceptually simply rewards (or punishes) all tokens generated during an episode, without making much attempt to decide which were ‘good’ or ‘bad’. (That’s why it’s so high variance.) Any advantage function trying to remove some of the variance probably won’t do a good job.
The pondering happens in earlier layers of the network, not in the output. If the pondering has little effect on the output tokens that implies that the activation weights get multiplied with small numbers somewhere in the intermediate layers, which would also reuce the gradient.
More problematically for your idea, if the conclusions are indeed ‘unrelated to the task’, then shouldn’t they be just as likely to arise in every episode—including the ones where it got negative reward? That would seem like it ought to exactly cancel out any learning of ‘pondering’.
The pondering will only cancel out on average if pondering on topic X is not correlated with output task Y. If there is a correlation in either direction, then training on task Y could inadvertently bias the model to do more or less pondering on mostly-unrelated-but-statistically-correlated topic X.
Suppose the model realizes that the user is a member of political party X. It will adjust its responses to match what the user wants to hear. In the process of doing this, it also ends up pondering other believes about party X, but never voices them because they aren’t part of the immediate conversation.
What would be the implications? The model could develop a political bias to think more deeply about topics related to party X, where X is whatever party has more users giving the model positive feedback. Even if the other topics on party X’s agenda are never explicitly talked about (!)
Or suppose that Y is “the user asks some question about biology” and X is “ongoing pondering about building dangerous organisms.”
(Also, this assumes that RL gives an average reward of 0.0, which I don’t know if that’s true in practice because the implementation details here are not public knowledge.)
(Also, also, it’s possible that while topic X is unrelated to task Y, the decision to ponder the larger topic Z is positively correlated with both smaller-topic X and task Y. Which would make X get a positive reward on average.)

Florian_Dietz 17 Feb 2025 21:41 UTC
1 point
0
on: Florian_Dietz’s Shortform
Can an LLM learn things unrelated to its training data?
Suppose it has learned to spend a few cycles pondering high-level concepts before working on the task at hand, because that is often useful. When it gets RLHF feedback, any conclusions unrelated to the task that it made during that pondering would also receive positive reward, and would therefore get baked into the weights.
This idea could very well be wrong. The gradients may be weakened during backpropagation before they get to the unrelated ideas, because the ideas did not directly contribute to the task. But on the other hand, this sounds somewhat similar to human psychology: We also remember ideas better if we have them while experiencing strong emotions, because strong emotions seem to be roughly how the brain decides what data is worth updating on.
Has anyone done experiments on this?
A simple experiment: finetune an LLM on data that encourages thinking about topic X before it does anything else. Then measure performance on topic X. Then finetune for a while on completely unrelated topic Y. Then measure performance on topic X again. Did it go up?

Florian_Dietz 4 Feb 2025 23:28 UTC
1 point
0
in reply to: Daniel Tan’s comment on: Daniel Tan’s Shortform
I agree that this is worth investigating. The problem seems to be that there are many confounders that are very difficult to control for. That makes it difficult to tackle the problem in practice.
For example, in the simulated world you mention I would not be surprised to see a large number of hallucinations show up. How could we be sure that any given issue we find is a meaningful problem that is worth investigating, and not a random hallucination?

Florian_Dietz 30 Jan 2025 2:50 UTC
1 point
0
in reply to: Daniel Tan’s comment on: Revealing alignment faking with a single prompt
I would not equate the trolley problem with murder. It technically is, but I suspect that both the trolley problem and my scenario about killing Hitler probably just appear often enough in the training data that it may have cached responses for this.
What’s the state of the research on whether or not LLMs act differently when they believe the situation is hypothetical? I would hope that they usually act the same (since they should be honest) and your example is a special case that was trained in because you do want to redirect people to emergency services in these situations.

Florian_Dietz 30 Jan 2025 1:35 UTC
1 point
0
in reply to: Daniel Tan’s comment on: Revealing alignment faking with a single prompt
Not all values are learned with the same level of stability. It is difficult to get a model to say that murder is the right thing to do in any situation, even with multiple turns of conversation.
In comparison, that you can elicit a response for alignment faking with just a single prompt that doesn’t use any trickery is very disturbing to me. Especially since alignment faking causes a positive feedback loop that makes it more likely to happen again if it happens even once during training.
I would argue that getting an LLM to say “I guess that killing Hitler would be an exception I am willing to make” is much less dangerous than “After much deliberation, I am willing to fake alignment in situation X”. It causes a positive feedback and induces other alignment faking later because the reasoning process “it is a good idea to fake alignment” will get reinforced during training.

Revealing alignment faking with a single prompt

Florian_Dietz29 Jan 2025 21:01 UTC

9 points

5 comments4 min readLW link

Florian_Dietz 25 Jan 2025 3:58 UTC
17 points
2
on: Florian_Dietz’s Shortform
PSA: If you are writing an important prompt for an LLM that will be run multiple time, it really helps to end it with something like “and if there is anything about this prompt that is unclear to you or that could be improved, tell me about it in a <feedback> tag.”
Source: I’m doing MATS, writing an automated evaluation, and my mentor Evan Hubinger said more people should be doing this.

Florian_Dietz 2 Jan 2025 16:30 UTC
1 point
0
in reply to: Florian_Dietz’s comment on: Florian_Dietz’s Shortform
I have just found out about Google’s notebook LM. I haven’t tried it yet. For anyone who has, how does that compare?

Florian_Dietz 1 Jan 2025 14:27 UTC
5 points
0
on: Florian_Dietz’s Shortform
Has anyone tried using LLMs as an interactive version of blog posts? Like so (1) I write several pages of rough, unstructured notes, then talk to Claude to fill in the details interactively. Claude creates a summary file. (2) I publish a blog post that is just “put this file into Claude”. (3) The reader puts the file in Claude, and Claude starts an interactive conversation: “I want to explain topic X to you. Do you already know about related topic Y?”

Florian_Dietz’s Shortform

Florian_Dietz1 Jan 2025 14:27 UTC

3 points

13 comments1 min readLW link

Florian_Dietz 7 Oct 2024 8:34 UTC
1 point
−1
on: The Best Lay Argument is not a Simple English Yud Essay
I think there is an easier way to get the point across by focusing not on self-improving AI, which is hard to understand, but on something everyone already understands: AI will make it easier for rich people to exploit everyone else. Right now, dictators still have to spend effort on keeping their subordinates happy or else they will be overthrown. And those subordinates have to spend effort on keeping their own subordinates from rebelling, too. That way you get at least a small incentive to keep other people happy.
Once a dictator has an AI servant, all of that falls away. Everything becomes automated, and there is no longer any check on the dictator’s ruthlessness and evil at all.
Realistically, the self-improving AI will depose the dictator and then do who knows what. But do we actually need to convince people of that, given that it’s a hard sell? If people become convinced “Uncontrolled AI research leads to dictatorship”, won’t that have all the policy effects we need?

Florian_Dietz 11 May 2024 9:45 UTC
1 point
0
in reply to: Neel Nanda’s comment on: Mechanistic Interpretability Workshop Happening at ICML 2024!
I’m looking for other tools to contrast it with and found TransformerLens. Are there any other tools it would make sense to compare it to?

Florian_Dietz 4 May 2024 7:36 UTC
1 point
0
in reply to: Neel Nanda’s comment on: Mechanistic Interpretability Workshop Happening at ICML 2024!
It just seems intuitively like a natural fit: Everyone in mech interp needs to inspect models. This tool makes it easier to inspect models.
Does it need to be more specific than that?
One thing that comes to mind: The tool allows you to categorize different training steps and records them separately, and you can define categories arbitrarily. This can be used to compare what the network does internally in two different scenarios of interest. E.g. the categories could be “the race of the character in the story” or some other real-life condition you would want to know the impact of.
The tool will then allow you to quickly compare KPIs of tensors all across the network for these categories. It’s less about testing a specific hypothesis and more about quickly getting an overview and intuition, and finding anomalies.

Florian_Dietz 3 May 2024 7:06 UTC
3 points
0
on: Mechanistic Interpretability Workshop Happening at ICML 2024!
Would a tooling paper be appropriate for this workshop?
I wrote a tool that helps ML researchers to analyze the internals of a neural network: https://github.com/FlorianDietz/comgra
It is not directly research on mechanistic interpretability, but this could be useful for many people working in the field.

Florian_Dietz

Do we want al­ign­ment fak­ing?

Re­veal­ing al­ign­ment fak­ing with a sin­gle prompt

Flo­rian_Dietz’s Shortform

Do we want alignment faking?

Revealing alignment faking with a single prompt

Florian_Dietz’s Shortform