Archimedes comments on StefanHex’s Shortform

Archimedes 12 Feb 2025 5:34 UTC
1 point
0
FVU_B doesn’t make sense but I don’t see where you’re getting FVU_B from.

Here’s the code I’m seeing:
```
resid_sum_of_squares = (
    (flattened_sae_input - flattened_sae_out).pow(2).sum(dim=-1)
)
total_sum_of_squares = (
    (flattened_sae_input - flattened_sae_input.mean(dim=0)).pow(2).sum(-1)
)

mse = resid_sum_of_squares / flattened_mask.sum()
explained_variance = 1 - resid_sum_of_squares / total_sum_of_squares
```
Explained variance = 1 - FVU = 1 - (residual sum of squares) / (total sum of squares)
- StefanHex 12 Feb 2025 10:19 UTC
  2 points
  0
  Parent
  I think this is the sum over the vector dimension, but not over the samples. The sum (mean) over samples is taken later in this line which happens after the division
```
        metrics[f"{metric_name}"] = torch.cat(metric_values).mean().item()
```
  - Archimedes 13 Feb 2025 0:47 UTC
    1 point
    0
    Parent
    Let’s suppose that’s the case. I’m still not clear on how are you getting to FVU_B?
    - StefanHex 13 Feb 2025 9:33 UTC
      2 points
      0
      Parent
      The previous lines calculate the ratio (or 1-ratio) stored in the “explained variance” key for every sample/batch. Then in that later quoted line, the list is averaged, I.e. we”re taking the sample average over the ratio. That’s the FVU_B formula.
      
      Let me know if this clears it up or if we’re misunderstanding each other!