Connor Kissane comments on Sparse Autoencoders Work on Attention Layer Outputs

[Connor Kissane]

Thanks for the comment! We always use the pre-ReLU feature activation, which is equal to the post-ReLU activation (given that the feature is activate), and is purely linear function of z. Edited the post for clarity.

[Ali Shehper]

I see. Thanks for the clarification!