We might also expect these circuits to take into account relative position rather than absolute position, especially using sinusoidal rather than learned positional encodings.
An interesting approach would be to encode the key and query values in a way that deliberately removes positional dependence (for example, run the base model twice with randomly offset positional encodings, train the key/​query value to approximate one encoding from the other) then incorporate a relative positional dependence into the learned large QK pair dictionary.
We might also expect these circuits to take into account relative position rather than absolute position, especially using sinusoidal rather than learned positional encodings.
An interesting approach would be to encode the key and query values in a way that deliberately removes positional dependence (for example, run the base model twice with randomly offset positional encodings, train the key/​query value to approximate one encoding from the other) then incorporate a relative positional dependence into the learned large QK pair dictionary.