Although, tokenized features are dissimilar to normal features in that they don’t vary in activation strength. Tokenized features are either 0 or 1 (or norm of the vector). So it’s not exactly an apples-to-apples comparison w/ a similar sized dictionary of normal SAE features, although that plot would be nice!
Although, tokenized features are dissimilar to normal features in that they don’t vary in activation strength. Tokenized features are either 0 or 1 (or norm of the vector). So it’s not exactly an apples-to-apples comparison w/ a similar sized dictionary of normal SAE features, although that plot would be nice!