I give a counterargument to this in the typo-riddled, poorly-written Tweet here. Sadly I won’t have a chance to write up thoughts here more cleanly for a few days.
ETA: Briefly, the key points are:
Honesty issues for introspection aren’t obviously much worse than they are for simple probing. (But fair if you’re already not excited about simple probing.)
When you can ask models arbitrary questions about their cognition, I think it’s probably quite difficult for a model to tell on which inputs it can get away with lying.
Probably I misunderstood your concern. I interpreted your concern about settings where we don’t have access to ground truth as relating to cases where the model could lie about its inner states without us being able to tell (because of lack of ground truth). But maybe you’re more worried about being able to develop a (sufficiently diverse) introspection training signal in the first place?
I’ll also note that I’m approaching this from the angle of “does introspection have worse problems with lack-of-ground-truth than traditional interpretability?” where I think the answer isn’t that clear without thinking about it more. Traditional interpretability often hill-climbs on “producing explanations that seem plausible” (instead of hill climbing on ground-truth explanations, which we almost never have access to), and I’m not sure whether this poses more of a problem for traditional interpretability vs. black-box approaches like introspection.
I give a counterargument to this in the typo-riddled, poorly-written Tweet here. Sadly I won’t have a chance to write up thoughts here more cleanly for a few days.
ETA: Briefly, the key points are:
Honesty issues for introspection aren’t obviously much worse than they are for simple probing. (But fair if you’re already not excited about simple probing.)
When you can ask models arbitrary questions about their cognition, I think it’s probably quite difficult for a model to tell on which inputs it can get away with lying.
I’m confused/skeptical about this being relevant, I thought honesty is orthogonal to whether the model has access to its mental states.
Probably I misunderstood your concern. I interpreted your concern about settings where we don’t have access to ground truth as relating to cases where the model could lie about its inner states without us being able to tell (because of lack of ground truth). But maybe you’re more worried about being able to develop a (sufficiently diverse) introspection training signal in the first place?
I’ll also note that I’m approaching this from the angle of “does introspection have worse problems with lack-of-ground-truth than traditional interpretability?” where I think the answer isn’t that clear without thinking about it more. Traditional interpretability often hill-climbs on “producing explanations that seem plausible” (instead of hill climbing on ground-truth explanations, which we almost never have access to), and I’m not sure whether this poses more of a problem for traditional interpretability vs. black-box approaches like introspection.
Thanks Sam. That tweet could be a good stand-alone LW post once you have time to clean up.