Why I’m not too worried about architecture-dependent mech interp methods:
I’ve heard people argue that we should develop mechanistic interpretability methods that can be applied to any architecture. While this is certainly a nice-to-have, and maybe a sign that a method is principled, I don’t think this criterion itself is important.
I think that the biggest hurdle for interpretability is to understand any AI that produces advanced language (>=GPT2 level). We don’t know how to write a non-ML program that speaks English, let alone reason, and we have no idea how GPT2 does it. I expect that doing this the first time is going to be significantly harder, than doing this the 2nd time. Kind of how “understand an Alien mind” is much harder than “understand the 2nd Alien mind”.
Edit: Understanding an image model (say Inception V1 CNN) does feel like a significant step down, in the sense that these models feel significantly less “smart” and capable than LLMs.
Agreed. A related thought is that we might only need to be able to interpret a single model at a particular capability level to unlock the safety benefits, as long as we can make a sufficient case that we should use that model. We don’t care inherently about interpreting GPT-4, we care about there existing a GPT-4 level model that we can interpret.
I’ve heard people argue that we should develop mechanistic interpretability methods that can be applied to any architecture.
I think the usual reason this claim is made is because the person making the claim thinks it’s very plausible LLMs aren’t the paradigm that lead to AGI. If that’s the case, then interpretability that’s indexed heavily on them gets us understanding of something qualitatively weaker than we’d like. I agree that there’ll be some transfer, but it seems better and not-very-hard to talk about how well different kinds of work transfer.
Why I’m not too worried about architecture-dependent mech interp methods:
I’ve heard people argue that we should develop mechanistic interpretability methods that can be applied to any architecture. While this is certainly a nice-to-have, and maybe a sign that a method is principled, I don’t think this criterion itself is important.
I think that the biggest hurdle for interpretability is to understand any AI that produces advanced language (>=GPT2 level). We don’t know how to write a non-ML program that speaks English, let alone reason, and we have no idea how GPT2 does it. I expect that doing this the first time is going to be significantly harder, than doing this the 2nd time. Kind of how “understand an Alien mind” is much harder than “understand the 2nd Alien mind”.
Edit: Understanding an image model (say Inception V1 CNN) does feel like a significant step down, in the sense that these models feel significantly less “smart” and capable than LLMs.
Agreed. A related thought is that we might only need to be able to interpret a single model at a particular capability level to unlock the safety benefits, as long as we can make a sufficient case that we should use that model. We don’t care inherently about interpreting GPT-4, we care about there existing a GPT-4 level model that we can interpret.
I think the usual reason this claim is made is because the person making the claim thinks it’s very plausible LLMs aren’t the paradigm that lead to AGI. If that’s the case, then interpretability that’s indexed heavily on them gets us understanding of something qualitatively weaker than we’d like. I agree that there’ll be some transfer, but it seems better and not-very-hard to talk about how well different kinds of work transfer.