Probably I misunderstood your concern. I interpreted your concern about settings where we don’t have access to ground truth as relating to cases where the model could lie about its inner states without us being able to tell (because of lack of ground truth). But maybe you’re more worried about being able to develop a (sufficiently diverse) introspection training signal in the first place?
I’ll also note that I’m approaching this from the angle of “does introspection have worse problems with lack-of-ground-truth than traditional interpretability?” where I think the answer isn’t that clear without thinking about it more. Traditional interpretability often hill-climbs on “producing explanations that seem plausible” (instead of hill climbing on ground-truth explanations, which we almost never have access to), and I’m not sure whether this poses more of a problem for traditional interpretability vs. black-box approaches like introspection.
Probably I misunderstood your concern. I interpreted your concern about settings where we don’t have access to ground truth as relating to cases where the model could lie about its inner states without us being able to tell (because of lack of ground truth). But maybe you’re more worried about being able to develop a (sufficiently diverse) introspection training signal in the first place?
I’ll also note that I’m approaching this from the angle of “does introspection have worse problems with lack-of-ground-truth than traditional interpretability?” where I think the answer isn’t that clear without thinking about it more. Traditional interpretability often hill-climbs on “producing explanations that seem plausible” (instead of hill climbing on ground-truth explanations, which we almost never have access to), and I’m not sure whether this poses more of a problem for traditional interpretability vs. black-box approaches like introspection.