One hypothesis for how transformers generate text is that they calculate semantically meaningful primitives in early layers of the residual stream, which are converted to a high-level execution plan in middle layers, followed by concrete tokens in the final layers.
Is there any empirical evidence for this? Or is this just a general observation?
Is there any empirical evidence for this? Or is this just a general observation?