See here for an updated version of my thinking based on discussion with Daniel:
abramdemski
Karma: 19,367
Why Don’t We Just… Shoggoth+Face+Paraphraser?
Yeah, I agree that’s at least somewhat true. So, given that, is it a good move or not? I don’t see much of an upside, since there’s a heavy financial incentive for the big labs to do little about concerning results observed in such models. IE, when it comes to the question of whether frontier labs training o1-like models should follow OpenAI’s example of avoiding safety training for the CoT, I think it’s right to discourage them from doing this rather than encourage them… although, I say this with very low confidence.
I don’t think it is worth getting into all the stuff you’re mentioning here, but I think a key crux is that I’m expecting the face to be quite dumb (e.g. literally 3.5 sonnet might suffice).
I’m reacting to this whole thing as a possible default configuration going forward, rather than as it exists today. All of my concerns are about what happens when you scale it up. For example, I don’t find o1′s current level of deception hugely concerning in its own right; rather, I see it as a bad sign for the technology as it continues to develop.
I guess one way of framing it is that I find the shoggoth/face idea great as a science experiment; it gives us useful evidence! However, it doesn’t make very much sense to me as a safety method intended for deployment.
Ah, yep, this makes sense to me.
Yeah, this is a good point, which doesn’t seem addressed by any idea so far.
Chess is like a bounded, mathematically described universe where all the instrumental convergence stays contained, and only accomplishes a very limited instrumentality in our universe (IE chess programs gain a limited sort of power here by being good playmates).
LLMs touch on the real world far more than that, such that MCTS-like skill at navigating “the LLM world” in contrast to chess sounds to me like it may create a concerning level of real-world-relevant instrumental convergence.
Still, IMO, exploiting the frozen planner through adversarial inputs in a single step seems pretty unlikely to be a fruitful strategy for the optimized planner. If the optimized planner is simply trying to accurately convey information to the frozen planner, probably >99% of that information is best to convey through human-understandable text.
Well, I’m not sure. As you mention, it depends on the step size. It also depends on how vulnerable to adversarial inputs LLMs are and how easy they are to find. I haven’t looked into the research on this, but it sounds empirically checkable. If there are lots of adversarial inputs which have a wide array of impacts on LLM behavior, then it would seem very plausible that the optimized planner could find useful ones without being specifically steered in that direction.
This is especially likely to hold under the following conditions:
We can also combine this with other proposals, such as paraphrasing.
Yeah, you’re right, I no longer think it’s an interesting proposal.
I’m talking about training only the Face, not training the policy (shoggoth) at all with the proposal I’m imagining.
And, these should clearly be separate models such that the training of one doesn’t generalize to the other.
So, making the Face more deceptive doesn’t kill the canary?
Ah, yes, I had neglected this important distinction.
So what do you do in the end, throw away the face?
It seems worth pointing out that although the CoT isn’t trained to be deceptive, it is trained to think thoughts which help out a deceptive face. So it seems plausible that a multi-step training process like:
GPT → basic chat training → shoggoth+face training → discard face & align shoggoth
Would result in deceptive capabilities: the shoggoth CoT learns to explicitly reason about what helps deceive the alignment training in the shoggoth+face step, and then in the final step, some of this is destroyed but some is driven underground. IE, learning to explicitly reason about deception creates latent structure which is useful for that task, and this latent structure will somewhat transfer to the case where the deception has to be hidden from the CoT.
The question is whether we end up with a highly misaligned policy (and this is clear from the chain of thought such that we can discover this). I think Daniel was expecting in the baseline proposal that you end up with a Face that does at least some censoring/mild deception due to training incentives (I think we already see a bit of this with o1).
So it sounds like we have some disagreement about what degree of deception is considered problematic. Again, I’m worried about basin-of-attraction type dynamics. We can think of a whole frontier lab as collectively being a sort of self-improving AGI; we want that recursive self-improvement to be going down a good path rather than a bad one. The LLMs are already helping to improve themselves in little ways (EG as coding assistants) and big ways (EG constitutional AI). So it’s an environment where seeds of deceptiveness can grow and take some control of the overall trajectory.
Maybe your concern is that it is plausible that the policy won’t be misaligned, but the Face will be and this will screw us over. This seems plausible and is worth worrying about (with both proposals). However, I thought that a core part of the hope is that the Face isn’t smart enough to cleverly scheme (unless this schemeing happens in the CoT itself). And indeed, if the Face can cleverly scheme without CoT, what is stopping the policy from doing this too?
I’m not sure what you’re getting at here. It seems like the idea is that Face + CoT is, together, a clever schemer—in the o1 report, we already see cases of explicit scheming in the CoT. So then, using the Face+CoT together (EG trying to train it further, using it in supervision steps for later iterations of the system, etc) is dangerous and difficult for the same reason that a single deceptive LLM would be. If we use Face+CoT together, we have to reason about them together in that way. Or, if we are using them as separate pieces, then we have to account for the part of the overall deceptive-schemer strategy they’ve learned.
I guess you could use Face+CoT as a supervisor (EG in constitutional AI) by looking at the CoT (both by hand and using some automation) and throwing out any deceptive examples, so as to not use them for supervision.
But the overall goal should be to produce a highly non-deceptive system in the end, I think.
Another idea would be to do the whole Face+CoT thing, and then filter for honesty on a bunch of examples, and then re-train a whole new system (well, you can start from the same GPT) only on those honest examples.
Assuming the goals are done over say 1-10 year timescales, or maybe even just 1 year timescales with no reward-shaping/giving feedback for intermediate rewards at all, I do think that the system won’t work well enough to be relevant, since it requires way too much time training, and plausibly way too much compute depending on how sparse the feedback actually is.
Ah, I wasn’t thinking “sparse” here meant anywhere near that sparse. I thought your dense-vs-sparse was doing something like contrasting RLHF (very dense, basically no instrumental convergence) with chess (very sparse, plenty of instrumental convergence).
I still think o1 is moving towards chess on this spectrum.
Async is fine with me. Please feel encouraged to start it.
My current epistemic state is pretty “I’m not sure at all, it’s a complicated subject” fwiw. I’ll try to generate some responses anyway.
make the system overall much harder to shape in ways you want (because it is actively hiding things from you so your RL is less accurate)
I thought the point was that we wouldn’t train against the CoT basically at all with the baseline proposal? (Similar to o1.)
Sure. Therefore, you can’t check for deceptive behavior using the CoT when you’re trying to train against deceptive behavior. Therefore, Hubinger et al’s conclusions about how deceptive behavior makes the system harder to train still hold even if you sorta get a transparency tool out of this.
You can still inspect the underlying CoT to know if you have big issues.
If other companies go in this direction like OpenAI has, then they’ll probably hide the CoT from the end-users for similar reasons, so you can’t say that the upside is that you, the end user, get to see whether the model is deliberately deceiving you with respect to specific questions.
So you probably mean the upside is that the companies can see this info in the CoT, to check whether models are generally behaving in a deceptive way (similar to what we saw in the o1 report).
Sure, but you’re making a commitment not to use this in training, so what are you doing with this information? If the model is deceptive, you just throw it away (or, more realistically, make excuses about how the deception you observed isn’t so bad). But your version of the proposal deliberately trains the model to be deceptive. So yeah, you can look, but what’s the point? You’re checking whether a coin you rigged to land on heads in fact landed on heads.
(My guess is that you have this concern with the baseline protocol and thus don’t like moving further in this direction.)
Yeah, you’re right about that. But there is a difference. I can think of the original proposal as a “canary in a coal mine”: you hope that the model doesn’t learn deceptive behavior (you plan to chuck it out if you detect any), but you still follow Daniel’s division of labor & refuse to do any alignment training on the CoT so that you have a chance of detecting deception if it occurs.
On the other hand, your version of Daniel’s protocol loses this property. Metaphorically, it’s like killing the canary before you go into the coal mine. If the canary was supposed to be an early warning to get out of there, you’ve completely defeated the point, and for what? (This is the part I’m unclear about, and why I asked my original question to you, rather than Daniel.)
Deceptive behavior is hard to remove once it creeps in. This is nasty stuff. A technique for getting deceptive behavior to reveal itself in the CoT would be wonderfully useful if you could then use that in the training signal to remove the deceptive behavior, but the proposal here relies on not doing that.
Basically, if there is a basin of attraction for corrigibility, it seems to me that deceptive behavior is the opposite of that. It puts you outside the basin of attraction. It means you’re on-coarse for Evan Hubinger’s story about how malign inner optimizers develop. It means you’re on-coarse for the Bad Ending rather than the Good Ending. And for what?
My take (very much siding with Evan Hubinger here) is that the target should be keeping well away from deceptive behavior throughout training, so that there’s never that kernel of skill that could grow into a malign inner optimizer.
Would you be interested in having a (perhaps brief) LW Dialogue about it where you start with a basic intro to your shoggoth/mask division-of-labor, and I then present my critique?
My point here is that at the capability level of GPT4, this distinction isn’t very important. There’s no way to know for sure until it is too late, of course, but it seems pretty plausible that GPT4 isn’t cleverly scheming. It is merely human-level at deception, and doesn’t pursue any coherent overarching goal with it. It clumsily muddles through with mildly-above-average-for-human convincingness. For most queries (it seems plausible to me) it isn’t even adequately coherent to make a crisp distinction between whether it’s honestly trying to answer the question vs deceptively trying to make an answer look good; at its level of capability, it’s mostly the same thing one way or the other. The exceptions to this “mostly” aren’t strategic enough that we expect them to route around obstacles cleverly.
It isn’t much, but it is more than I naively expected.
I think the crux is I think that the important parts of of LLMs re safety isn’t their safety properties specifically, but rather the evidence they give to what alignment-relevant properties future AIs have
[insert standard skepticism about these sorts of generalizations when generalizing to superintelligence]
But what lesson do you think you can generalize, and why do you think you can generalize that?
I think this is a crux, in that I don’t buy o1 as progressing to a regime where we lose so much dense feedback that it’s alignment relevant, because I think sparse-feedback RL will almost certainly be super-uncompetitive with every other AI architecture until well after AI automates all alignment research.
So, as a speculative example, further along in the direction of o1 you could have something like MCTS help train these things to solve very difficult math problems, with the sparse feedback being given for complete formal proofs.
Similarly, playing text-based video games, with the sparse feedback given for winning.
Similarly, training CoT to reason about code, with sparse feedback given for predictions of the code output.
Etc.
You think these sorts of things just won’t work well enough to be relevant?
I think you’re mostly right about the world, but I’m going to continue to articulate disagreements based on my sense of dissatisfaction. You should probably mostly read me as speaking from a should-world rather than being very confused about the is-world.
The bitter lesson is insufficient to explain the lack of structure we’re seeing. I gave the example of Whisper. I haven’t actually used Whisper, so correct me if I’m wrong—maybe there is a way to get more nuanced probabilistic information out of it? But the bitter lesson is about methods of achieving capabilities, not about the capabilities themselves. Producing a plain text output rather than a more richly annotated text that describes some of the uncertainty is a design choice.
To give another example, LLMs could learn a conditional model that’s annotated with metadata like author, date and time, etc. Google Lambda reportedly had something like author-vectors, so that different “characters” could be easily summoned. I would love to play with that, EG, averaging the author-vectors of specific authors I’m interested in to see what it’s like to mix their voices. In theory, LLMs could also learn to predict the metadata. You could use partially-labeled-data approaches such as the EM algorithm to train LLMs to predict the metadata labels while also training them to use those labels in text generation. This would give a rich set of useful capabilities. This would be scientifically useful, too: predictions of date, author, and other metadata would be interesting for historians.
In this way, we could actually get closer to a “truth machine”. There’s a world of difference between the kinds of inferences you can make from ChatGPT’s opinion about a text vs inferences you could make by treating these as legit latent variables and seeing what latent-variable-inference algorithms think.
To give a third example, there’s Drexler’s Quasilinguistic Neural Representations.
You say “it’s a matter of demand”. So why does it feel so much like big tech is eager to push the chatbot model on everyone? Everyone is scared to fall behind in the AI race, ever since ChatGPT made it feel plausible that AI would drive significant market share. But did big tech make the correct guess about what there was demand for? Or did everyone over-update on the specific form of ChatGPT, and now there’s just not that much else out there to reveal what the demand really is? Little sparkle-emoji buttons decorate everything; press to talk to the modern equivalent of Clippy.
Process supervision would be helpful, particularly if the model is punished for sensitive dependence on paraphrasings of the CoT, or rewarded for producing CoT which is useful even after paraphrasing.
One of the concerns that I have is that (I believe) the LLM before the CoT-training will be sensitive to some non-human-readable patterns, similar to adversarial examples for image recognition. The CoT training will encourage any patterns which help reach the desired conclusions, whether those patterns are human-readable or not.
A lot of possible process-supervision procedures will not do anything to help prevent this. For example, rewarding CoT for being human-understandable won’t help, because the reward signal there can miss the subtle patterns being used. It only avoids the outcome where the CoT looks non-human-readable; it doesn’t avoid the outcome where the CoT actually contains a bunch of human-unreadable patterns.
Using paraphrasing to avoid this is heavily dependent on the strength of the paraphraser. The paraphraser needs to have really good coverage of alternative ways to convey the same semantic information, so that any non-human-readable patterns get randomized. The paraphraser itself can’t be vulnerable to steering by adversarial inputs.
I more-or-less agree with Eliezer’s comment (to the extent that I have the data necessary to evaluate his words, which is greater than most, but still, I didn’t know him in 1996). I have a small beef with his bolded “MIRI is always in every instance” claim, because a universal like that is quite a strong claim, and I would be very unsurprised to find a single counterexample somewhere (particularly if we include every MIRI employee and everything they’ve ever said while employed at MIRI).
What I am trying to say is something looser and more gestalt. I do think what I am saying contains some disagreement with some spirit-of-MIRI, and possibly some specific others at MIRI, such that I could say I’ve updated on the modern progress of AI in a different way than they have.
For example, in my update, the modern progress of LLMs points towards the Paul side of some Eliezer-Paul debates. (I would have to think harder about how to spell out exactly which Eliezer-Paul debates.)
One thing I can say is that I myself often argued using “naive misinterpretation”-like cases such as the paperclip example. However, I was also very aware of the Eliezer-meme “the AI will understand what the humans mean, it just won’t care”. I would have predicted difficulty in building a system which correctly interprets and correctly cares about human requests to the extent that GPT4 does.
This does not mean that AI safety is easy, or that it is solved; only that it is easier than I anticipated at this particular level of capability.
Getting more specific to what I wrote in the post:
My claim is that modern LLMs are “doing roughly what they seem like they are doing” and “internalize human intuitive concepts”. This does include some kind of claim that these systems are more-or-less ethical (they appear to be trying to be helpful and friendly, therefore they “roughly are”).
The reason I don’t think this contradicts with Eliezer’s bolded claim (“Getting a shape into the AI’s preferences is different from getting it into the AI’s predictive model”) is that I read Eliezer as talking about strongly superhuman AI with this claim. It is not too difficult to get something into the values of some basic reinforcement learning agent, to the extent that something like that has values worth speaking of. It gets increasingly difficult as the agent gets cleverer. At the level of intelligence of, say, GPT4, there is not a clear difference between getting the LLM to really care about something vs merely getting those values into its predictive model. It may be deceptive or honest; or, it could even be meaningless to classify it as deceptive or honest. This is less true of o1, since we can see it actively scheming to deceive.
In my view, the misaligned behavior is split across the shoggoth and face in most cases. In the case which Daniel K anticipates, the Shoggoth is misaligned in the sense of explicitly (but honestly) plotting to deceive, while the face is misaligned in the sense of dishonestly concealing.
So I am interpreting you as asking why we might expect that specific division of responsibilities, rather than some other divisions such as those you mention.
One reason is that we already see this empirically in the o1 report.
However, to me, the distinction between your case 1 and 2 is not so important. Both scenarios you propose preserve the important thing: there is a mismatch between an honest CoT and a dishonest final answer, which would allow the deceptive final answer to be detected.
There could be other cases where this doesn’t happen; this method is not a guaranteed lie detector. To me, the point is that it will at least sometimes happen. A lie detector that only sometimes works is better than no lie detector at all.
If you give a kid a diary and try to avoid giving them any reason to think you’re reading it, then the kid might write honestly in the diary and reveal some deceptions you wouldn’t previously have known about. The kid might also do any number of other things. However, setting aside the ethics of snooping on your kid and lying to them about it, it’s better than nothing.