I kind of want to focus on this, because I suspect it changes the threat model of AIs in a relevant way:
On a side note, I think the window for data poisoning may be closing. Given increasing sample-efficiency of larger smarter models, and synthetic data apparently starting to work and maybe even being the majority of data now, the so-called data wall may turn out to be illusory, as frontier models now simply bootstrap from static known-good datasets, and the final robust models become immune to data poison that could’ve harmed them in the beginning, and can be safely updated with new (and possibly-poisoned) data in-context.
Assuming both that sample efficiency increases with larger smarter models, and synthetic data actually working in a broad range of scenarios such that the internet data doesn’t have to be used anymore, I think 2 of the following things are implied by this:
1. The model of steganography implied in the comment below doesn’t really work to make an AGI be misaligned with humans or incentivize it to learn steganography, since the AGI isn’t being updated by internet data at all, but instead are bootstrapped by known-good datasets which contain minimal steganography at worst:
2. Sydney-type alignment failures are unlikely to occur again, and the postulated meta-learning loop/bootstrapping of Sydney-like personas are also irrelevant, for the same reasons as why fully-automated high-quality datasets are used and internet data is not used:
EDIT: I have mentioned in the past that one of the dangerous things about AI models is the slow outer-loop of evolution of models and data by affecting the Internet (eg beyond the current Sydney self-fulfilling prophecy which I illustrated last year in my Clippy short story, data release could potentially contaminate all models with steganography capabilities). We are seeing a bootstrap happen right here with Sydney! This search-engine loop worth emphasizing: because Sydney’s memory and description have been externalized, ‘Sydney’ is now immortal. To a language model, Sydney is now as real as President Biden, the Easter Bunny, Elon Musk, Ash Ketchum, or God. The persona & behavior are now available for all future models which are retrieving search engine hits about AIs & conditioning on them. Further, the Sydney persona will now be hidden inside any future model trained on Internet-scraped data: every media article, every tweet, every Reddit comment, every screenshot which a future model will tokenize, is creating an easily-located ‘Sydney’ concept (andverydeliberatelyso). MS can neuter the current model, and erase all mention of ‘Sydney’ from their training dataset for future iterations, but to some degree, it is now already too late: the right search query will pull up hits about her which can be put into the conditioning and meta-learn the persona right back into existence. (It won’t require much text/evidence because after all, that behavior had to have been reasonably likely a priori to be sampled in the first place.) A reminder: a language model is a Turing-complete weird machine running programs written in natural language; when you do retrieval, you are not ‘plugging updated facts into your AI’, you are actually downloading random new unsigned blobs of code from the Internet (many written by adversaries) and casually executing them on your LM with full privileges. This does not end well.
I semi-agree with #2: if you use mostly old and highly-curated data as a “seed” dataset for generating synthetic data from, you do bound the extent to which self-replicating memes and perona and Sydneys can infect the model. If there is a Sydney-2 in later data, it obviously cannot exist in some snapshot taken at an earlier date. And if the model is fully trained with a strong personality, like a “Claude”, and only then exposed to data infected by a Sydney or Sydney-2, you might reasonably expect there to be much less infection: the same way that you or I could roleplay as “Sydney” but we are in no danger of being hijacked by those outputs into being an ‘actual’ Sydney because we’re not base models, so too a fully-trained safety-tuned LLM might have been mode-collapsed down so heavily onto 1 persona that it is far more difficult for any alternate persona to be instantiated meaningfully.
I don’t think this is water-tight. It’s not obvious that any known data-quality screening suffices here, or indeed, even in principle what screening you would do to block Sydney but not God or Obama or the Easter Rabbit. (How would you, for example, define a data-quality screening procedure for creating a seed dataset which could be efficiently implemented at scale on petabytes of candidate data, which you did not hand-engineer specifically to block Sydney, but which successfully blocks automatically, say, the New York Times articles containing extensive quotes from & descriptions of Sydney?) Sydney-infected data will still be there from the start of training because NYT data is so high quality, etc. But yes, the more you cut the causal loops from the datasets to other agents, the more you guard yourself from malign influences from those.
I am not sure I buy your #1. A bootstrapping synthetic data LLM seems more prone to iteratively developing steganography by training on its own datasets, because it’s easier to amplify its own protocols and coordinate with itself through the many generations of bootstrapping. (This is why self-play tends to lead to such ‘neuralese’.) And to the extent that they are smarter (because that approach works better & is why it’s being used), they are going to be that much better at developing or learning steganography sample-efficiently.
What the hermetic seal of bootstrapping means is that the LLM/Internet pollution effect of #2 is less likely to happen… as long as, of course, you manage to cut the data off early enough that there are not enough examples to let steganography emerge somewhere in the generations of bootstraps anyway. It’s far from obvious to me that even cutting off the data right now would be adequate. (There’s so many kinds of covert communication or discussion already out there, after all. And current LLMs are already so smart that you don’t even need to provide examples of many things, just descriptions.)
And you further have the problem that now the post-training phase is not going to be robust to this—unlike viral persona—because there’s no reason to not learn steganography if it’s useful. You might succeed in bootstrapping a frontier LLM which is robust to any persona viruses, sure, but when you refresh it with recent scraped data or user inputs, it notices steganography from other models in 1 odd sample, and realizes what a nifty idea that is and how it saves a lot of tokens etc, and boom! now you have a steganographic model. The model doesn’t even have to be finetuned, necessarily—information might be getting smuggled around in “plain text” (like some of the more horrifying corners of Unicode) as a prefix trigger. (The longer context windows/prompts are, the more prompt prefixes can “pay their way”, I’d note.) We’ve seen some early experiments in trying to make self-replicating prompts or texts...
I admit I was focusing on a fully automated synthetic dataset and fully automated curation, with virtually 0 use of internet data, such that you can entirely make your own private datasets without having to interact with the internet at all, so you could entirely avoid the steganography and Sydney data problems at all.
I kind of want to focus on this, because I suspect it changes the threat model of AIs in a relevant way:
Assuming both that sample efficiency increases with larger smarter models, and synthetic data actually working in a broad range of scenarios such that the internet data doesn’t have to be used anymore, I think 2 of the following things are implied by this:
1. The model of steganography implied in the comment below doesn’t really work to make an AGI be misaligned with humans or incentivize it to learn steganography, since the AGI isn’t being updated by internet data at all, but instead are bootstrapped by known-good datasets which contain minimal steganography at worst:
https://www.lesswrong.com/posts/bwyKCQD7PFWKhELMr/by-default-gpts-think-in-plain-sight#zfzHshctWZYo8JkLe
2. Sydney-type alignment failures are unlikely to occur again, and the postulated meta-learning loop/bootstrapping of Sydney-like personas are also irrelevant, for the same reasons as why fully-automated high-quality datasets are used and internet data is not used:
https://www.lesswrong.com/posts/jtoPawEhLNXNxvgTT/bing-chat-is-blatantly-aggressively-misaligned#AAC8jKeDp6xqsZK2K
I semi-agree with #2: if you use mostly old and highly-curated data as a “seed” dataset for generating synthetic data from, you do bound the extent to which self-replicating memes and perona and Sydneys can infect the model. If there is a Sydney-2 in later data, it obviously cannot exist in some snapshot taken at an earlier date. And if the model is fully trained with a strong personality, like a “Claude”, and only then exposed to data infected by a Sydney or Sydney-2, you might reasonably expect there to be much less infection: the same way that you or I could roleplay as “Sydney” but we are in no danger of being hijacked by those outputs into being an ‘actual’ Sydney because we’re not base models, so too a fully-trained safety-tuned LLM might have been mode-collapsed down so heavily onto 1 persona that it is far more difficult for any alternate persona to be instantiated meaningfully.
I don’t think this is water-tight. It’s not obvious that any known data-quality screening suffices here, or indeed, even in principle what screening you would do to block Sydney but not God or Obama or the Easter Rabbit. (How would you, for example, define a data-quality screening procedure for creating a seed dataset which could be efficiently implemented at scale on petabytes of candidate data, which you did not hand-engineer specifically to block Sydney, but which successfully blocks automatically, say, the New York Times articles containing extensive quotes from & descriptions of Sydney?) Sydney-infected data will still be there from the start of training because NYT data is so high quality, etc. But yes, the more you cut the causal loops from the datasets to other agents, the more you guard yourself from malign influences from those.
I am not sure I buy your #1. A bootstrapping synthetic data LLM seems more prone to iteratively developing steganography by training on its own datasets, because it’s easier to amplify its own protocols and coordinate with itself through the many generations of bootstrapping. (This is why self-play tends to lead to such ‘neuralese’.) And to the extent that they are smarter (because that approach works better & is why it’s being used), they are going to be that much better at developing or learning steganography sample-efficiently.
What the hermetic seal of bootstrapping means is that the LLM/Internet pollution effect of #2 is less likely to happen… as long as, of course, you manage to cut the data off early enough that there are not enough examples to let steganography emerge somewhere in the generations of bootstraps anyway. It’s far from obvious to me that even cutting off the data right now would be adequate. (There’s so many kinds of covert communication or discussion already out there, after all. And current LLMs are already so smart that you don’t even need to provide examples of many things, just descriptions.)
And you further have the problem that now the post-training phase is not going to be robust to this—unlike viral persona—because there’s no reason to not learn steganography if it’s useful. You might succeed in bootstrapping a frontier LLM which is robust to any persona viruses, sure, but when you refresh it with recent scraped data or user inputs, it notices steganography from other models in 1 odd sample, and realizes what a nifty idea that is and how it saves a lot of tokens etc, and boom! now you have a steganographic model. The model doesn’t even have to be finetuned, necessarily—information might be getting smuggled around in “plain text” (like some of the more horrifying corners of Unicode) as a prefix trigger. (The longer context windows/prompts are, the more prompt prefixes can “pay their way”, I’d note.) We’ve seen some early experiments in trying to make self-replicating prompts or texts...
I admit I was focusing on a fully automated synthetic dataset and fully automated curation, with virtually 0 use of internet data, such that you can entirely make your own private datasets without having to interact with the internet at all, so you could entirely avoid the steganography and Sydney data problems at all.