Oliver Clive-Griffin comments on StefanHex’s Shortform

[Oliver Clive-Griffin]

This was really helpful, thanks! just wanting to clear up my understanding:

This is the wikipedia entry for FVU:

where:

There’s no mention of norms because (as I understand) $y$ and $\hat{y}$ are assumed to be scalar values so $S{S}_{\text{err}}$ and $S{S}_{\text{tot}}$ are scalar. Do I understand it correctly that you’re treating $\parallel x_n-x_{n,\text{pred}}{\parallel }^2$ as the multi-dimensional equivalent of $S{S}_{\text{err}}$ and $\parallel x_n-\mu {\parallel }^2$ as the multi-dimensional equivalent of $S{S}_{\text{tot}}$? This would make sense as using the squared norms of the differences makes it basis /​ rotation invariant.

[StefanHex]

Yep, that’s the generalisation that would make most sense

[Oliver Clive-Griffin]

Thanks. Also, in the case of crosscoders, where you have multiple output spaces, do you have any thoughts on the best way to aggregate across these? currently I’m just computing them separately and taking the mean. But I could see imagine it perhaps being better to just concat the spaces and do fvu on that, using l2 norm of the concated vectors.

[StefanHex]

Yeah you probably shouldn’t concat the spaces due to things like “they might have very different norms & baseline variances”. Maybe calculate each layer separately, then if they’re all similar average them together, otherwise keep separate and quote as separate numbers in your results