@Connor Kissane @Neel Nanda Does SAE work on MLP blocks of GPT2-small as well? I find the recovery rate significantly low (40%) for MLP activations of larger models like GPT2-small.
We’ve found slightly worse results for MLPs, but nowhere near 40%, I expect you’re training your SAEs badly. What exact metric equals 40% here?
@Connor Kissane @Neel Nanda Does SAE work on MLP blocks of GPT2-small as well? I find the recovery rate significantly low (40%) for MLP activations of larger models like GPT2-small.
We’ve found slightly worse results for MLPs, but nowhere near 40%, I expect you’re training your SAEs badly. What exact metric equals 40% here?