That’s very cool, I’m looking forward to seeing those results! The Top-K extension is particularly interesting, as that was something I wasn’t sure how to approach.
I imagine you’ve explored important directions I haven’t touched like better benchmarking, top-k implementation, and testing on larger models. Having multiple independent validations of an approach also seems valuable.
I’d be interested in continuing this line of research, especially circuits with Matryoshka SAEs. I’d love to hear about what directions you’re thinking of. Would you want to have a call sometime about collaboration or coordination? (I’ll DM you!)
That’s very cool, I’m looking forward to seeing those results! The Top-K extension is particularly interesting, as that was something I wasn’t sure how to approach.
I imagine you’ve explored important directions I haven’t touched like better benchmarking, top-k implementation, and testing on larger models. Having multiple independent validations of an approach also seems valuable.
I’d be interested in continuing this line of research, especially circuits with Matryoshka SAEs. I’d love to hear about what directions you’re thinking of. Would you want to have a call sometime about collaboration or coordination? (I’ll DM you!)
Really looking forward to reading your post!