dkirmani comments on Real-Time Research Recording: Can a Transformer Re-Derive Positional Info?

[dkirmani]

guessing this wouldn’t work without causal attention masking

[Neel Nanda]

Yeah, I think that’s purely symmetric.