Logan Riggs comments on Decomposing the QK circuit with Bilinear Sparse Dictionary Learning

[Logan Riggs]

Is there code available for this?

I’m mainly interested in the loss fuction. Specifically from footnote 4:

We also need to add a term to capture the interaction effect between the key-features and the query-transcoder bias, but we omit this for simplicity

I’m unsure how this is implemented or the motivation.