If we can tell what an AGI is thinking about, but not exactly what its thoughts are, will this be useful? Doesn’t a human-level intelligence need to be able to think about dangerous topics, in the course of doing useful cognition?
I think that most people doing mechanistic-ish interp would agree with this. Being able to say, ‘the Golden Gate Bridge feature is activated’ on its own isn’t that practically useful. This sort of work is the foundation for more sophisticated interp work that looks at compositions of features, causal chains, etc. But being able to cleanly identify features is a necessary first step for that. It would have been possible to go further beyond that by now if it hadn’t turned out that individual neurons don’t correspond well to features; that necessitated this sort of work on identifying features as a linear combination of multiple neurons.
That doesn’t mean, of course, that those directions won’t encounter blockers, or that this approach scales in a straightforward way past human-level. But I don’t think many people are thinking of this kind of single-feature identification as a solution to alignment; it’s an important step toward a larger goal.
Probably a much better way of getting a sense of the long-term agenda than reading my comment is to look back at Chris Olah’s “Interpretability Dreams” post.
Our present research aims to create a foundation for mechanistic interpretability research. In particular, we’re focused on trying to resolve the challenge of superposition. In doing so, it’s important to keep sight of what we’re trying to lay the foundations for. This essay summarizes those motivating aspirations – the exciting directions we hope will be possible if we can overcome the present challenges.
I think that most people doing mechanistic-ish interp would agree with this. Being able to say, ‘the Golden Gate Bridge feature is activated’ on its own isn’t that practically useful. This sort of work is the foundation for more sophisticated interp work that looks at compositions of features, causal chains, etc. But being able to cleanly identify features is a necessary first step for that. It would have been possible to go further beyond that by now if it hadn’t turned out that individual neurons don’t correspond well to features; that necessitated this sort of work on identifying features as a linear combination of multiple neurons.
One recent paper that starts to look at causal chains of features and is a useful pointer to the sort of direction (I expect) this research can go next is ‘Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models’; you might find it of interest.
That doesn’t mean, of course, that those directions won’t encounter blockers, or that this approach scales in a straightforward way past human-level. But I don’t think many people are thinking of this kind of single-feature identification as a solution to alignment; it’s an important step toward a larger goal.
Probably a much better way of getting a sense of the long-term agenda than reading my comment is to look back at Chris Olah’s “Interpretability Dreams” post.