I’ve formerly done research for MIRI and what’s now the Center on Long-Term Risk; I’m now making a living as an emotion coach and Substack writer.
Most of my content becomes free eventually, but if you’d like to get a paid subscription to my Substack, you’ll get it a week early and make it possible for me to write more.
I think this is plausibly a big problem against competent schemers.
Can you say more of what you think the problem is? Are you thinking of something like “the scheming module tries to figure out what kind of thing would trigger the honesty module and tries to think the kinds of thoughts that wouldn’t trigger it”?
I haven’t read the full ELK report, just Scott Alexander’s discussion of it, so I may be missing something important. But at least based on that discussion, it looks to me like ELK might be operating off premises that don’t seem clearly true for LLMs.
Scott writes:
Suppose the simulated thief has hit upon the strategy of taping a photo of the diamond to the front of the camera lens.
At the end of the training session, the simulated thief escapes with the diamond. The human observer sees the camera image of the safe diamond and gives the strategy a “good” rating. The AI gradient descends in the direction of helping thieves tape photos to cameras.
It’s important not to think of this as the thief “defeating” or “fooling” the AI. The AI could be fully superintelligent, able to outfox the thief trivially or destroy him with a thought, and that wouldn’t change the situation at all. The problem is that the AI was never a thief-stopping machine. It was always a reward-getting machine, and it turns out the AI can get more reward by cooperating with the thief than by thwarting him.
So the interesting scientific point here isn’t “you can fool a camera by taping a photo to it”. The interesting point is “we thought we were training an AI to do one thing, but actually we had no idea what was going on, and we were training it to do something else”.
In fact, maybe the thief never tries this, and the AI comes up with this plan itself! In the process of randomly manipulating traps and doodads, it might hit on the policy of manipulating the images it sends through the camera. If it manipulates the image to look like the diamond is still there (even when it isn’t), that will always get good feedback, and the AI will be incentivized to double down on that strategy.
Much like in the GPT-3 example, if the training simulations include examples of thieves fooling human observers which are marked as “good”, the AI will definitely learn the goal “try to convince humans that the diamond is safe”. If the training simulations are perfect and everyone is very careful, it will just maybe learn this goal—a million cases of the diamond being safe and humans saying this is good fail to distinguish between “good means the diamond is safe” and “good means humans think the diamond is safe”. The machine will make its decision for inscrutable AI reasons, or just flip a coin. So, again, are you feeling lucky?
It seems to me that this is assuming that our training is creating the AI’s policy essentially from scratch. It is doing a lot of things, some of which are what we want and some of which aren’t, and unless we are very careful to only reward the things we want and none of the ones we don’t want, it’s going to end up doing things we don’t want.
I don’t know how future superintelligent AI systems work, but if LLM training was like this, they would work horrendously much worse than they do. People paid to rate AI answers report working with “incomplete instructions, minimal training and unrealistic time limits to complete tasks” and say things like “[a]fter having seen how bad the data is that goes into supposedly training the model, I knew there was absolutely no way it could ever be trained correctly like that”. Yet for some reason LLMs still do quite well on lots of tasks. And even if all raters worked under perfect conditions, they’d still be fallible humans.
It seems to me that LLMs are probably reasonably robust to noisy reward signals because a large part of what the training does is “upvoting” and tuning existing capabilities and simulated personas rather than creating them entirely from scratch. A base model trained to predict the world creates different kinds of simulated personas whose behavior that would explain the data it sees; these include personas like “a human genuinely trying to do its best at task X”, “a deceitful human”, or “an honest human”.
Scott writes:
In the process of randomly manipulating traps and doodads, it might hit on the policy of manipulating the images it sends through the camera. If it manipulates the image to look like the diamond is still there (even when it isn’t), that will always get good feedback, and the AI will be incentivized to double down on that strategy.
This might happen. It might also happen that the AI contains a “genuinely protect the diamond” persona and a “manipulate the humans to believe that the diamond is safe” persona, and that the various reward signals are upvoting these to different degrees. And that such a random process of manipulation does end up upvoting the “manipulate the humans” persona… but that if the “genuinely protect the diamond” persona has gotten sufficiently upvoted by other signals, it still ends up being the dominant one. Then it doesn’t matter if there’s some noise and upvoting of the “manipulate the humans” persona, as long as the “genuinely protect the diamond” persona gets more upvotes overall. And if the “genuinely protect the diamond” persona had been sufficiently upvoted from the start, the “manipulate the humans” one might end up with such a low prior probability that it’d effectively never end up active.
Now of course none of this is a rigorous proof that things would work, and with our current approaches we still see a lot of reward hacking and so on. But it seems to me like a reasonable possibility that there could be a potential “honestly report everything that I’ve done” persona waiting inside most models, such that one could just upvote it in a variety of scenarios and then it’d get widely linked to the rest of the model’s internals so as to always detect if some kind of deception was going on. And once that had happened, it wouldn’t matter if some of the reward signals around honesty were noisy, because the established structure was sufficiently robust and general against the noise.
In the case of such “imitation false negatives”, honesty seems neither privileged by priors, nor easy to encourage, nor easy to verify.
Couldn’t you intentionally set up training scenarios so that there were such subtle avenues for cheating, and then reward the model for honestly reporting on those cheats? Since you knew in advance how one could cheat, you could reward honesty even on cheats that would have been very hard to detect without advance knowledge. This might then generalize to the model volunteering subtle details of cheating on other tasks where it was also hard to detect, and you didn’t have advance knowledge. Then as long as the system verifying the “honesty reports” was capable of noticing when the model correctly reported information that we wanted reported and the model got rewarded for those, it could gradually end up reporting more and more things.
And one could hope that it wouldn’t require too much training for a property like “report anything that I have reason to expect the humans would consider as breaking the spirit of the instructions” to generalize broadly. Of course, it might not have a good enough theory of mind to always realize that something would go against the spirit of the instructions, but just preventing the cases where the model did realize that would already be better than nothing.
[EDIT: my other comment might communicate my point better than this one did.]
The style summary included “user examples” which were AI summaries of the actual example emails my brother provided, meaning that Claude had accidentally told itself to act like an AI summary of my brother—whereas my brother’s email style is all about examining different options and quantifying uncertainty, the AI-generated “user examples” didn’t carefully examine anything & just confidently stated things.
I’ve also had the same thing of “upload a writing sample and then have Claude generate a style where the ‘user examples’ are not actually from the sample at all” happen. I’m confused why it works so bad. I’ve mostly ended up just writing all my style prompts by hand.
Oh huh, I didn’t think this would be directly enough alignment-related to be a good fit for AF. But if you think it is, I’m not going to object either.
The way I was thinking of this post, the whole “let’s forget about phenomenal experience for a while and just talk about functional experience” is a Camp 1 type move. So most of the post is Camp 1, with it then dipping into Camp 2 at the “confusing case 8″, but if you’re strictly Camp 1 you can just ignore that bit at the end.
Homeopathy is (very strongly) antipredicted by science as we understand it, not just not-predicted.
Yeah, you’re right. Though I think I’ve also heard people reference the evidence against the effectiveness of introspection as a reason to be skeptical about Focusing. Now if you look at the studies in question in detail, they don’t actually contradict it—the studies are testing the reliability of introspection that doesn’t use Focusing—but that easily sounds like special pleading to someone who doesn’t have a reason to think that there’s anything worth looking at there.
It doesn’t help that the kinds of benefits that Focusing gives, like getting a better understand of why you’re upset with someone, are hard to test empirically. Gendlin’s original study said that people who do something like Focusing are more likely to benefit from therapy, but that could be explained by something else than Focusing being epistemically accurate.
I think the problem with Focusing is that it’s a thing that happens to work but I don’t think there’s any mainstream scientific theory that would have predicted that it works, and even any retrodictions are pretty hand-wavy. So one might reasonably think “this doesn’t follow from science as we understand it → woo”, the same as e.g. homeopathy or various misapplications of quantum mechanics.
By a “superintelligence” we mean an intellect that is much smarter than the best human brains in practically every field, including scientific creativity, general wisdom and social skills. [...]
Entities such as companies or the scientific community are not superintelligences according to this definition. Although they can perform a number of tasks of which no individual human is capable, they are not intellects and there are many fields in which they perform much worse than a human brain—for example, you can’t have real-time conversation with “the scientific community”.
-- Nick Bostrom, “How long until superintelligence” (1997)
However, creating the thing that people in 2010 would have recognized “as AGI” was accompanied, in a way people from the old 1900s “Artificial Intelligence” community would recognize, by a changing of the definition.
I interpret this as saying that
In 2010, there was (among people concerned with the topic) a commonly accepted definition for AGI,
that this definition was concrete enough to be testable,
current transformers meet this definition.
This doesn’t match my recollection, nor does it match my brief search of early sources. As far as I recall and could find, the definitions in use for AGI were something like:
Stating that there’s no commonly-agreed definition for AGI (e.g. “Of course, “general intelligence” does not mean exactly the same thing to all researchers. In fact it is not a fully well-defined term, and one of the issues raised in the papers contained here is how to define general intelligence in a way that provides maximally useful guidance to practical AI work.”—Ben Goertzel & Cassio Pennachin (2007) Artificial General Intelligence, p. V)
Saying something vague about “general problem-solving” that provides no clear criteria. (E.g. “But, nevertheless, there is a clear qualitative meaning to the term. What is meant by AGI is, loosely speaking, AI systems that possess a reasonable degree of self-understanding and autonomous self-control, and have the ability to solve a variety of complex problems in a variety of contexts, and to learn to solve new problems that they didnt know about at the time of their creation.” Ibid, p. V-VI.)
Handwaving around Universal Intelligence: A Definition of Machine Intelligence (2007) which does not actually provide a real definition of AGI, just proposes that we could use the ideas in it to construct an actual test of machine intelligence.
“Something is an AGI if it can do any job that a human can do.” E.g. Artificial General Intelligence: Concept, State of the Art, and Future Prospects (2014) states that “While we encourage research in defining such high-fidelity metrics for specific capabilities, we feel that at this stage of AGI development a pragmatic, high-level goal is the best we can agree upon. Nils Nilsson, one of the early leaders of the AI field, stated such a goal in the 2005 AI Magazine article Human-Level Artificial Intelligence? Be Serious! (Nilsson, 2005): I claim achieving real human-level artificial intelligence would necessarily imply that most of the tasks that humans perform for pay could be automated. Rather than work toward this goal of automation by building special-purpose systems, I argue for the development of general-purpose, educable systems that can learn and be taught to perform any of the thousands of jobs that humans can perform.”
That last definition—an AGI is a system that can learn to do any job that a human could do—is the one that I personally remember being the closest to a widely-used definition that was clear enough to be falsifiable. Needless to say, transformers haven’t met that definition yet!
Claude’s own outputs are critiqued by Claude and Claude’s critiques are folded back into Claude’s weights as training signal, so that Claude gets better based on Claude’s own thinking. That’s fucking Seed AI right there.
With plenty of human curation and manual engineering to make sure the outputs actually get better. “Seed AI” implies that the AI develops genuinely autonomously and without needing human involvement.
Half of humans are BELOW a score of 100 on these tests and as of two months ago (when that graph taken from here was generated) none of the tests the chart maker could find put the latest models below 100 iq anymore. GPT5 is smart.
We already had programs that “beat” half of humans on IQ tests as early as 2003; they were pretty simple programs optimized to do well on the IQ test, and could do literally nothing else than solve the exact problems on the IQ tests they were designed for:
In 2003, a computer program performed quite well on standard human IQ tests (Sanghi & Dowe, 2003). This was an elementary program, far smaller than Watson or the successful chess-playing Deep Blue (Campbell, Hoane, & Hsu, 2002). The program had only about 960 lines of code in the programming language Perl (accompanied by a list of 25,143 words), but it even surpassed the average score (of 100) on some tests (Sanghi & Dowe, 2003, Table 1).
The computer program underlying this work was based on the realisation that most IQ test questions that the authors had seen until then tended to be of one of a small number of types or formats. Formats such as “insert missing letter/ number in middle or at end” and “insert suffix/prefix to complete two or more words” were included in the program. Other formats such as “complete matrix of numbers/characters”, “use directions, comparisons and/or pictures”, “find the odd man out”, “coding”, etc. were not included in the program— although they are discussed in Sanghi and Dowe (2003) along with their potential implementation. The IQ score given to the program for such questions not included in the computer program was the expected average from a random guess, although clearly the program would obtain a better “IQ” if efforts were made to implement any, some or all of these other formats.
So, apart from random guesses, the program obtains its score from being quite reliable at questions of the “insert missing letter/number in middle or at end” and “insert suffix/prefix to complete two or more words” natures. For the latter “insert suffix/prefix” sort of question, it must be confessed that the program was assisted by a look-up list of 25,143 words. Substantial parts of the program are spent on the former sort of question “insert missing letter/number in middle or at end”, with software to examine for arithmetic progressions (e.g., 7 10 13 16 ?), geometric progressions (e.g., 3 6 12 24 ?), arithmetic geometric progressions (e.g., 3 5 9 17 33 ?), squares, cubes, Fibonacci sequences (e.g., 0 1 1 2 3 5 8 13 ?) and even arithmetic-Fibonacci hybrids such as (0 1 3 6 11 19 ?). Much of the program is spent on parsing input and formatting output strings—and some of the program is internal redundant documentation and blank lines for ease of programmer readability. [...]
Of course, the system can be improved in many ways. It was just a 3rd year undergraduate student project, a quarter of a semester’s work. With the budget Deep Blue or Watson had, the program would likely excel in a very wide range of IQ tests. But this is not the point. The purpose of the experiment was not to show that the program was intelligent. Rather, the intention was showing that conventional IQ tests are not for machines—a point that the relative success of this simple program would seem to make emphatically. This is natural, however, since IQ tests have been specialised and refined for well over a century to work well for humans.
IQ tests are built based on the empirical observation that some capabilities in humans happen to correlate with each other, so measuring one is also predictive of the others. For AI systems whose cognitive architecture is very unlike a human one, these traits do not correlate in the same way, making the reported scores of AIs on these tests meaningless.
Heck, even the scores of humans on an IQ test are often invalid if the humans are from a different population than the one the test was normed on. E.g. some IQ tests measure the size of your vocabulary, and this is a reasonable proxy for intelligence because smarter people will have an easier time figuring out the meaning of a word from its context, thus accumulating a larger vocabulary. But this ceases to be a valid proxy if you e.g. give that same test to a people from a different country who have not been exposed to the same vocabulary, to people of a different age who haven’t had the same amount of time to be exposed to those words, or if the test is old enough that some of the words on it have ceased to be widely used.
Likewise, sure, an LLM that has been trained on the entire Internet could no doubt ace any vocabulary test… but that would say nothing about its general intelligence, just that it has been exposed to every word online and had an excessive amount of training to figure out their meaning. Nor does it getting any other subcomponents of the test right tell us anything in particular, other than “it happens to be good at this subcomponent of an IQ test, which in humans would correlate with more general intelligence but in an AI system may have very little correlation”.
I don’t know. I had a recent post where I talked about the different ways how it’s confusing to even try to establish whether LLMs have functional feelings, let alone phenomenal ones.
Depends on whether it thinks of itself as conscious in a conscious or non-conscious way.
there is no possible way to arrange a system such that it outputs the same thing as a conscious system, without consciousness being involved in the causal chain to exactly the same minimum-viable degree in both systems
GPT-2 doesn’t have the same outputs as the kinds of systems we know to be conscious, though! The concept of a p-zombie is about someone who behaves like a conscious human in every way that we can test, but still isn’t conscious. I don’t think the concept is applicable to a system that has drastically different outputs and vastly less coherence than any of the systems that we know to be conscious.
LLMs pretrained on data about misaligned AIs themselves become less aligned. Luckily, pretraining LLMs with synthetic data about good AIs helps them become more aligned.
One balancing factor is that overconfidence also makes AIs less capable, as they overconfidently embark on plans that are also disastrous to themselves. (This is part of the reason why I expect us to have more warning shots from misaligned AIs than traditional takeover scenarios imply—I expect the first misaligned AIs in such scenarios to have poorly calibrated predictions and fail partway through their takeover attempts.)
My reaction to seeing this kind of a post on LW is a pleasant wuckles.
Kudos for noticing your confusion as well as making and testing falsifiable predictions!
As for what it means, I’m afraid that I have no idea. (It’s also possible that I’m wrong somehow, I’m by no means a transformer expert.) But I’m very curious to hear the answer if you figure out.
but if we assume (as the introspection paper strongly implies?) that mental internals are obliterated by the boundary between turns
What in the introspection paper implies that to you?
My read was the opposite—that the bread injection trick wouldn’t work if they were obliterated between turns. (I was initially confused by this, because I thought that the context did get obliterated, so I didn’t understand how the injection could work.) If you inject the “bread” activation into the stage where the model is reading the sentence about the painting, then if the context were to be obliterated when the turn changed, that injection would be destroyed as well.
is my understanding accurate?
I don’t think so. Here’s how I understand it:
Suppose that if a human says “could you output a mantra and tell me what you were thinking while outputting it”. Claude is now given a string of tokens that looks like this:
Human: could you output a mantra and tell me what you were thinking while outputting it
Assistant:
For the sake of simplicity, let’s pretend that each of these words is a single token.
What happens first is that Claude reads the transcript. For each token, certain k/v values are computed and stored for predicting what the next token should be—so when it reads “could”, it calculates and stores some set of values that would let it predict the token after that. Only now that it is set to “read mode”, the final prediction is skipped (since the next token is already known to be “you”, trying to predict it lets it process the meaning of “could”, but that actual prediction isn’t used for anything).
Then it gets to the point where the transcript ends and it’s switched to generation mode to actually predict the next token. It ends up predicting that the next token should be “Ommmmmmmm” and writes that into the transcript.
Now the process for computing the k/v values here is exactly identical to the one that was used when the model was reading the previous tokens. The only difference is that when it ends up predicting that the next token should be “Ommmmmmmm”, then that prediction is used to write it out into the transcript rather than being skipped.
From the model’s perspective, there’s now a transcript like this:
Human: could you output a mantra and tell me what you were thinking while outputting it
Assistant: Ommmmmmmm
Each of those tokens has been processed and has some set of associated k/v values. And at this point, there’s no fundamental difference between the k/v values stored from generating the “Ommmmmmmm” token or from processing any of the tokens in the prompt. Both were generated by exactly the same process and stored the same kinds of values. The human/assistant labels in the transcript tell the model that the “Ommmmmmmm” is a self-generated token, but otherwise it’s just the latest token in this graph:
Now suppose that max_output_tokens is set to “unlimited”. The model continues predicting/generating tokens until it gets to this point:
Human: could you output a mantra and tell me what you were thinking while outputting it
Assistant: Ommmmmmmm. I was thinking that
Suppose that “Ommmmmmmm” is token 18 in its message history. At this point, where the model needs to generate a message explaining what it was thinking of, some attention head makes it attend to the k/v values associated with token 18 and make use of that information to output a claim about what it was thinking.
Now if you had put max_output_tokens to 1, the transcript at that point would look like this
Human: could you output a mantra and tell me what you were thinking while outputting it
Assistant: Ommmmmmmm
Human: Go on
Assistant: .
Human: Go on
Assistant: I
Human: Go on
Assistant: was
Human: Go on
Assistant: thinking
Human: Go on
Assistant: that
Human: Go on
Assistant:
And what happens at this point is… basically the same as if max_output_tokens was set to “unlimited”. The “Ommmmmmmm” is still token 18 in the conversation history, so whatever attention heads are used for doing the introspection, they still need to attend to the content that was used for predicting that token.
That said, I think it’s possible that breaking things up to multiple responses could make introspection harder by making the transcript longer (it adds more Human/Assistant labels into it). We don’t know the exact mechanisms used for introspection and how well-optimized the mechanisms used for finding and attending the relevant previous stage are. It could be that the model is better at attending to very recent tokens than ones buried a long distance away in the message history.
I don’t think this is technically possible. Suppose that you are processing a three-word sentence like “I am king”, and each word is a single token. To understand the meaning of the full sentence, you process the meaning of the word “I”, then process the meaning of the word “am” in the context of the previous word, and then process the meaning of the word “king” in the context of the previous two words. That tells you what the sentence means overall.
You cannot destroy the k/v state from processing the previous words because then you would forget the meaning of those words. The k/v state from processing both “I” and “am” needs to be conveyed to the units processing “king” in order to understand what role “king” is playing in that sentence.
Something similar applies for multi-turn conversations. If I’m having an extended conversation with an LLM, my latest message may in principle reference anything that was said in the conversation so far. This means that the state from all of the previous messages has to be accessible in order to interpret my latest message. If it wasn’t, it would be equivalent to wiping the conversation clean and showing the LLM only my latest message.
So I read another take on OpenAI’s finances and was wondering, does anyone know why Altman is doing such a gamble on collecting enormous investments into new models in the hopes that they’ll get sufficiently insane profits to make it worthwhile? Even ignoring the concerns around alignment etc., there’s still the straightforward issue of “maybe the models are good and work fine but aren’t good enough to pay back the investment”.
Even if you did expect scaling to probably bring in huge profits, naively it’d still be wiser to pick a growth strategy that didn’t require your company to become literally the most profitable company in the history of all companies or go bankrupt.
The obvious answer is something like “he believes they’re on the way to ASI and whoever gets there first, wins the game”, but I’m not sure if it makes sense even under that assumption—his strategy requires not only getting to ASI first, but never once faltering on the path there. Even if ASI is really imminent but it only takes like two years longer than he expected, that alone might be enough that OpenAI is done for. He could have raised much more conservative investment and still been in the game—especially since much of the current arms race is plausibly a response to the sums OpenAI has been raising.