StefanHex comments on StefanHex’s Shortform

[StefanHex]

I think this is the sum over the vector dimension, but not over the samples. The sum (mean) over samples is taken later in this line which happens after the division

        metrics[f"{metric_name}"] = torch.cat(metric_values).mean().item()

Edit: And to clarify, my impression is that people think of this as alternative definitions of FVU and you got to pick one, rather than one being right and one being a bug.

Edit2: And I’m in touch with the SAEBench authors about making a PR to change this /​ add both options (and by extension probably doing the same in SAELens); though I won’t mind if anyone else does it!

[Gurkenglas]

Ah, oops. I think I got confused by the absence of L_2 syntax in your formula for FVU_B. (I agree that FVU_A is more principled ^^.)

[StefanHex]

Oops, fixed!