eggsyntax comments on Transformers Represent Belief State Geometry in their Residual Stream

[eggsyntax]

One optimal predictor for this data would be to maintain a belief over which of the three HMM states we are in

As well as inferring the HMM itself from the data.