Anish Mudide comments on Efficient Dictionary Learning with Switch Sparse Autoencoders

[Anish Mudide]

Thanks for the question -- $f$ is calculated over an entire batch of inputs, not a single $x$. Figure 1 shows how the Switch SAE processes a single residual stream activation $x$.

[phenomanon]

Thank you for the answer, that makes more sense.