Oliver Sourbut comments on Transformers Represent Belief State Geometry in their Residual Stream

[Oliver Sourbut]

the original ‘theorem’ was wordcelled nonsense

Lol! I guess if there was a more precise theorem statement in the vicinity gestured, it wasn’t nonsense? But in any case, I agree the original presentation is dreadful. John’s is much better.

I would be curious to hear a *precise * statement why the result here follows from the Good Regular Theorem.

A quick go at it, might have typos.

Suppose we have

• $X$ (hidden) state

• $Y$ output/​observation

and a predictor

• $S$ (predictor) state

• $\hat{Y}$ predictor output

• $R$ the reward or goal or what have you (some way of scoring ‘was $\hat{Y}$ right?’)

with structure

$$\begin{array}{cc}X& \to Y\\ X& \to R\\ Y\to S& \to \hat{Y}\to R\end{array}$$

Then GR trivially says $S$ (predictor state) should model the posterior $P\left(X|Y\right)$.

Now if these are all instead processes (time-indexed), we have HMM

• $X_t$ (hidden) states

• $Y_t$ observations

and predictor process

• $S_t$ (predictor) states

• ${\hat{Y}}_t$ predictions

• $R_t$ rewards

with structure

$$\begin{array}{cc}{X}_t& \to X_{t+1}\\ X_t& \to Y_t\\ S_{t-1}& \to S_t\\ Y_t\to S_t& \to {\hat{Y}}_{t+1}\to R_{t+1}\\ Y_{t+1}& \to R_{t+1}\end{array}$$

Drawing together $(X_{t+1},Y_{t+1},{\hat{Y}}_{t+1},R_{t+1})$ as $G_t$ the ‘goal’, we have a GR motif

$$\begin{array}{cc}{X}_t& \to Y_t\\ Y_t& \to S_t\to G_t\\ S_{t-1}& \to S_t\\ X_t& \to G_t\end{array}$$

so $S_t$ must model $P(X_t|S_{t-1},Y_t)$; by induction that is $P(X_t|S_0,Y_1,...,Y_t)$.