I’m a bit confused why this happens, if the circuit only “needs” three layers of composition
I trained these models on only 22B tokens, of which only about 4B was Python code, and their residual stream has width 512. It totally wouldn’t surprise me if it just didn’;t have enough data or capacity in 3L, even though it was technically capable.
I trained these models on only 22B tokens, of which only about 4B was Python code, and their residual stream has width 512. It totally wouldn’t surprise me if it just didn’;t have enough data or capacity in 3L, even though it was technically capable.
Ah, that makes sense!