Therefore, the longer you interact with the LLM, eventually the LLM will have collapsed into a waluigi. All the LLM needs is a single line of dialogue to trigger the collapse.
This seems wrong. I think the mistake you’re making is when you argue that because there’s some chance X happens at each step and X is an absorbing state, therefore you have to end up at X eventually. However, this is only true if you assume the conclusion and claim that the prior probability of luigis is zero. If there is some prior probability of a luigi, each non-waluigi step increases the probability of never observing a transition to a waluigi a little bit.
Agreed. To give a concrete toy example: Suppose that Luigi always outputs “A”, and Waluigi is {50% A, 50% B}. If the prior is {50% luigi, 50% waluigi}, each “A” outputted is a 2:1 update towards Luigi. The probability of “B” keeps dropping, and the probability of ever seeing a “B” asymptotes to 50% (as it must).
This is the case for perfect predictors, but there could be some argument about particular kinds of imperfect predictors which supports the claim in the post.
Context windows could make the claim from the post correct. Since the simulator can only consider a bounded amount of evidence at once, its P[Waluigi] has a lower bound. Meanwhile, it takes much less evidence than fits in the context window to bring its P[Luigi] down to effectively 0.
Imagine that, in your example, once Waluigi outputs B it will always continue outputting B (if he’s already revealed to be Waluigi, there’s no point in acting like Luigi). If there’s a context window of 10, then the simulator’s probability of Waluigi never goes below 1/1025, while Luigi’s probability permanently goes to 0 once B is outputted, and so the simulator is guaranteed to eventually get stuck at Waluigi.
I expect this is true for most imperfections that simulators can have; its harder to keep track of a bunch of small updates for X over Y than it is for one big update for Y over X.
Yep I think you might be right about the maths actually.
I’m thinking that waluigis with 50% A and 50% B have been eliminated by llm pretraining and definitely by rlhf. The only waluigis that remain are deceptive-at-initialisation.
So what we have left is a superposition of a bunch of luigis and a bunch of waluigis, where the waluigis are deceptive, and for each waluigi there is a different phrase that would trigger them.
I’m not claiming basin of attraction is the entire space of interpolation between waluigis and luigis.
Actually, maybe “attractor” is the wrong technical word to use here. What I want to convey is that the amplitude of the luigis can only grow very slowly and can be reversed, but the amplitude of the waluigi can suddenly jump to 100% in a single token and would remain there permanently. What’s the right dynamical-systemy term for that?
Describing the waluigi states as stable equilibria and the luigi states as unstable equilibria captures most of what you’re describing in the last paragraph here, though without the amplitude of each.
I think your original idea was tenable. LLMs have limited memory, so the waluigi hypothesis can’t keep dropping in probability forever, since evidence is lost. The probability only becomes small—but this means if you run for long enough you do in fact expect the transition.
LLMs are high order Markov models, meaning they can’t really balance two different hypotheses in the way you describe; because evidence drops out of memory eventually, the probability of Waluigi drops very small instead of dropping to zero. This makes an eventual waluigi transition inevitable as claimed in the post.
You’re correct. The finite context window biases the dynamics towards simulacra which can be evidenced by short prompts, i.e. biases away from luigis and towards waluigis.
But let me be more pedantic and less dramatic than I was in the article — the waluigi transitions aren’t inevitable. The waluigi are approximately-absorbing classes in the Markov chain, but there are other approximately-absorbing classes which the luigi can fall into. For example, endlessly cycling through the same word (mode-collapse) is also an approximately-absorbing class.
“Open Problems in GPT Simulator Theory” (forthcoming)
Specifically, this is a chapter on the preferred basis problem for GPT Simulator Theory.
TLDR: GPT Simulator Theory says that the language model μ:Tk→Δ(T) decomposes into a linear interpolation μ=∑s∈Sαsμs where each μs:Tk→Δ(T) is a “simulacra” and the amplitudes as update in an approximately Bayesian way. However, this decomposition is non-unique, making GPT Simulator Theory either ill-defined, arbitrary, or trivial. By comparing this problem to the preferred basis problem in quantum mechanics, I construct various potential solutions and compare them.
The transform isn’t symmetric though right? A character portraying “good” behaviour is, narratively speaking, more likely to have been deceitful the whole time or transform into a villain than for the antagonist to turn “good”.
Each non-waluigi step increases the probability of never observing a transition to a waluigi a little bit.
Each non-Waluigi step increases the probability of never observing a transition to Waluigi a little bit, but not unboundedly so. As a toy example, we could start with P(Waluigi) = P(Luigi) = 0.5. Even if P(Luigi) monotonically increases, finding novel evidence that Luigi isn’t a deceptive Waluigi becomes progressively harder. Therefore, P(Luigi) could converge to, say, 0.8.
However, once Luigi says something Waluigi-like, we immediately jump to a world where P(Waluigi) = 0.95, since this trope is very common. To get back to Luigi, we would have to rely on a trope where a character goes from good to bad to good. These tropes exist, but they are less common. Obviously, this assumes that the context window is large enough to “remember” when Luigi turned bad. After the model forgets, we need a “bad to good” trope to get back to Luigi, and these are more common.
I disagree. The crux of the matter is the limited memory of an LLM. If the LLM had unlimited memory, then every Luigi act would further accumulate a little evidence against Waluigi. But because LLMs can only update on so much context, the probability drops to a small one instead of continuing to drop to zero. This makes waluigi inevitable in the long run.
I agree. Though is it just the limited context window that causes the effect? I may be mistaken, but from my memory it seems like they emerge sooner than you would expect if this was the only reason (given the size of the context window of gpt3).
This comment seems to rest on a dubious assumption. I think you’re saying:
The model has a distribution over a set of behaviors that includes “behave like luigi” and “behave like waluigi”. If there’s prior probability on “behave like luigi”, then in the limit of luigi-like steps, the posterior of “behave like luigi” goes to 1.
The first sentence is dubious though. Why would the LLM’s behavior come from a distribution over a space that includes “behave like luigi (forever)”? My question is informal, because maybe you can translate between distributions over [behaviors for all time] and [behaviors as functions from a history to a next action]. But these two representations seem to suggest different “natural” kinds of distributions. (In particular, a condition like non-dogmatism—not assigning probability 0 to anything in the space—might not be preserved by the translation.)
I think what the OP is saying is that each luigi step is actually a superposition step, and therefore each next line adds up the probability of collapse. However, from a pure trope perspective I believe this is not really the case—in most works of fiction that have a twist, the author tends to leave at least some subtle clues for the twist (luigi turning out to be a waluigi). So it is possible at least for some lines to decrease the possibility of waluigi collapse.
This seems wrong. I think the mistake you’re making is when you argue that because there’s some chance X happens at each step and X is an absorbing state, therefore you have to end up at X eventually. However, this is only true if you assume the conclusion and claim that the prior probability of luigis is zero. If there is some prior probability of a luigi, each non-waluigi step increases the probability of never observing a transition to a waluigi a little bit.
Agreed. To give a concrete toy example: Suppose that Luigi always outputs “A”, and Waluigi is {50% A, 50% B}. If the prior is {50% luigi, 50% waluigi}, each “A” outputted is a 2:1 update towards Luigi. The probability of “B” keeps dropping, and the probability of ever seeing a “B” asymptotes to 50% (as it must).
This is the case for perfect predictors, but there could be some argument about particular kinds of imperfect predictors which supports the claim in the post.
Context windows could make the claim from the post correct. Since the simulator can only consider a bounded amount of evidence at once, its P[Waluigi] has a lower bound. Meanwhile, it takes much less evidence than fits in the context window to bring its P[Luigi] down to effectively 0.
Imagine that, in your example, once Waluigi outputs B it will always continue outputting B (if he’s already revealed to be Waluigi, there’s no point in acting like Luigi). If there’s a context window of 10, then the simulator’s probability of Waluigi never goes below 1/1025, while Luigi’s probability permanently goes to 0 once B is outputted, and so the simulator is guaranteed to eventually get stuck at Waluigi.
I expect this is true for most imperfections that simulators can have; its harder to keep track of a bunch of small updates for X over Y than it is for one big update for Y over X.
Yep I think you might be right about the maths actually.
I’m thinking that waluigis with 50% A and 50% B have been eliminated by llm pretraining and definitely by rlhf. The only waluigis that remain are deceptive-at-initialisation.
So what we have left is a superposition of a bunch of luigis and a bunch of waluigis, where the waluigis are deceptive, and for each waluigi there is a different phrase that would trigger them.
I’m not claiming basin of attraction is the entire space of interpolation between waluigis and luigis.
Actually, maybe “attractor” is the wrong technical word to use here. What I want to convey is that the amplitude of the luigis can only grow very slowly and can be reversed, but the amplitude of the waluigi can suddenly jump to 100% in a single token and would remain there permanently. What’s the right dynamical-systemy term for that?
Describing the waluigi states as stable equilibria and the luigi states as unstable equilibria captures most of what you’re describing in the last paragraph here, though without the amplitude of each.
I think your original idea was tenable. LLMs have limited memory, so the waluigi hypothesis can’t keep dropping in probability forever, since evidence is lost. The probability only becomes small—but this means if you run for long enough you do in fact expect the transition.
LLMs are high order Markov models, meaning they can’t really balance two different hypotheses in the way you describe; because evidence drops out of memory eventually, the probability of Waluigi drops very small instead of dropping to zero. This makes an eventual waluigi transition inevitable as claimed in the post.
You’re correct. The finite context window biases the dynamics towards simulacra which can be evidenced by short prompts, i.e. biases away from luigis and towards waluigis.
But let me be more pedantic and less dramatic than I was in the article — the waluigi transitions aren’t inevitable. The waluigi are approximately-absorbing classes in the Markov chain, but there are other approximately-absorbing classes which the luigi can fall into. For example, endlessly cycling through the same word (mode-collapse) is also an approximately-absorbing class.
What report is the image pulled from?
“Open Problems in GPT Simulator Theory” (forthcoming)
Specifically, this is a chapter on the preferred basis problem for GPT Simulator Theory.
TLDR: GPT Simulator Theory says that the language model μ:Tk→Δ(T) decomposes into a linear interpolation μ=∑s∈Sαsμs where each μs:Tk→Δ(T) is a “simulacra” and the amplitudes as update in an approximately Bayesian way. However, this decomposition is non-unique, making GPT Simulator Theory either ill-defined, arbitrary, or trivial. By comparing this problem to the preferred basis problem in quantum mechanics, I construct various potential solutions and compare them.
The transform isn’t symmetric though right? A character portraying “good” behaviour is, narratively speaking, more likely to have been deceitful the whole time or transform into a villain than for the antagonist to turn “good”.
Each non-Waluigi step increases the probability of never observing a transition to Waluigi a little bit, but not unboundedly so. As a toy example, we could start with P(Waluigi) = P(Luigi) = 0.5. Even if P(Luigi) monotonically increases, finding novel evidence that Luigi isn’t a deceptive Waluigi becomes progressively harder. Therefore, P(Luigi) could converge to, say, 0.8.
However, once Luigi says something Waluigi-like, we immediately jump to a world where P(Waluigi) = 0.95, since this trope is very common. To get back to Luigi, we would have to rely on a trope where a character goes from good to bad to good. These tropes exist, but they are less common. Obviously, this assumes that the context window is large enough to “remember” when Luigi turned bad. After the model forgets, we need a “bad to good” trope to get back to Luigi, and these are more common.
I disagree. The crux of the matter is the limited memory of an LLM. If the LLM had unlimited memory, then every Luigi act would further accumulate a little evidence against Waluigi. But because LLMs can only update on so much context, the probability drops to a small one instead of continuing to drop to zero. This makes waluigi inevitable in the long run.
I agree. Though is it just the limited context window that causes the effect? I may be mistaken, but from my memory it seems like they emerge sooner than you would expect if this was the only reason (given the size of the context window of gpt3).
A good question. I’ve never seen it happen myself; so where I’m standing, it looks like short emergence examples are cherry-picked.
This comment seems to rest on a dubious assumption. I think you’re saying:
The first sentence is dubious though. Why would the LLM’s behavior come from a distribution over a space that includes “behave like luigi (forever)”? My question is informal, because maybe you can translate between distributions over [behaviors for all time] and [behaviors as functions from a history to a next action]. But these two representations seem to suggest different “natural” kinds of distributions. (In particular, a condition like non-dogmatism—not assigning probability 0 to anything in the space—might not be preserved by the translation.)
I think what the OP is saying is that each luigi step is actually a superposition step, and therefore each next line adds up the probability of collapse. However, from a pure trope perspective I believe this is not really the case—in most works of fiction that have a twist, the author tends to leave at least some subtle clues for the twist (luigi turning out to be a waluigi). So it is possible at least for some lines to decrease the possibility of waluigi collapse.