Maybe a better question would be—why didn’t these issues (lack of robust explanation) get in the way of the Steinhardt paper I linked? They were in fact able to execute something like the plan I sketch here: use vague understanding to guess which model components attend to features which are spuriously correlated with the thing you want, then use the rest of the model as an improved classifier for the thing you want.
My guess is that the classification task for waterbirds is sufficiently easy that butchering a substantial part of the model is fine. It won’t usually be viable to ablate everything that looks similar to an undesirable property. In some cases, this might be fine due to redundancy, but if there is heavy redundancy, I also expect that you’ve missed some stuff if you just look for components which look to a given target.
Not super high confidence overall.
Edit: it also seems likely to me that there is a more principled and simpler approach like using LEACE which works just as well or better (but I’m unsure and I’m not familiar with that paper or the literature here).
My guess is that the classification task for waterbirds is sufficiently easy that butchering a substantial part of the model is fine. It won’t usually be viable to ablate everything that looks similar to an undesirable property. In some cases, this might be fine due to redundancy, but if there is heavy redundancy, I also expect that you’ve missed some stuff if you just look for components which look to a given target.
Not super high confidence overall.
Edit: it also seems likely to me that there is a more principled and simpler approach like using LEACE which works just as well or better (but I’m unsure and I’m not familiar with that paper or the literature here).