Jett Janiak comments on Characterizing stable regions in the residual stream of LLMs

[Jett Janiak]

I believe there are two phenomena happening during training

1. Predictions corresponding to the same stable region become more similar, i.e. stable regions become more stable. We can observe this in the animations.

2. Existing regions split, resulting in more regions.

3. 

I hypothesize that

1. could be some kind of error correction. Models learn to rectify errors coming from superposition interference or another kind of noise.

2. could be interpreted as more capable models picking up on subtler differences between the prompts and adjusting their predictions.