Darn, exactly the project I was hoping to do at MATS! :-) Nice work!
There’s pretty suggestive evidence that the LLM first decides to refuse (and emits token’s like “I’m sorry”), then later writes a justification for refusing (see some of the hilarious reasons generated for not telling you how to make a teddy bear, after being activation engineered into refusing this). So I would view arguing anything about the nature of the refusal process from the text of the refusal-justification given afterwards as circumstantial evidence at best. But then you have direct gradient evidence that these directions matter, so I suppose the refusal texts you quote, if considered just as an argument as to why it’s sensible model behavior that that direction ought to matter (as opposed to evidence that it does), are helpful — however, I think you might want to make this distinction clearer in your write-up.
Looking through Latent 2213, my impression is that a) it mostly triggers on a wide variety of innocuous-looking tokens indicating the ends of phrases (so likely it’s summarizing those phrases), and b) those phrases tend to be about a legal, medical, or social process or chain of consequences causing something really bad to happen (e.g. cancer, sexual abuse, poisoning). This also rather fits with the set of latents that it has significant cosine similarity to. So I’d summarize it as “a complex or technically-involved process leading to a dramatically bad outcome”.
If that’s accurate, then it tending to trigger the refusal direction makes a lot of sense.
Darn, exactly the project I was hoping to do at MATS! :-)
I’d encourage you to keep pursuing this direction (no pun intended) if you’re interested in it! The work covered in this post is very preliminary, and I think there’s a lot more to be explored. Feel free to reach out, would be happy to coordinate!
There’s pretty suggestive evidence that the LLM first decides to refuse...
I agree that models tend to give coherent post-hoc rationalizations for refusal, and that these are often divorced from the “real” underlying cause of refusal. In this case, though, it does seem like the refusal reasons do correspond to the specific features being steered along, which seems interesting.
Darn, exactly the project I was hoping to do at MATS! :-) Nice work!
There’s pretty suggestive evidence that the LLM first decides to refuse (and emits token’s like “I’m sorry”), then later writes a justification for refusing (see some of the hilarious reasons generated for not telling you how to make a teddy bear, after being activation engineered into refusing this). So I would view arguing anything about the nature of the refusal process from the text of the refusal-justification given afterwards as circumstantial evidence at best. But then you have direct gradient evidence that these directions matter, so I suppose the refusal texts you quote, if considered just as an argument as to why it’s sensible model behavior that that direction ought to matter (as opposed to evidence that it does), are helpful — however, I think you might want to make this distinction clearer in your write-up.
Looking through Latent 2213, my impression is that a) it mostly triggers on a wide variety of innocuous-looking tokens indicating the ends of phrases (so likely it’s summarizing those phrases), and b) those phrases tend to be about a legal, medical, or social process or chain of consequences causing something really bad to happen (e.g. cancer, sexual abuse, poisoning). This also rather fits with the set of latents that it has significant cosine similarity to. So I’d summarize it as “a complex or technically-involved process leading to a dramatically bad outcome”.
If that’s accurate, then it tending to trigger the refusal direction makes a lot of sense.
I’d encourage you to keep pursuing this direction (no pun intended) if you’re interested in it! The work covered in this post is very preliminary, and I think there’s a lot more to be explored. Feel free to reach out, would be happy to coordinate!
I agree that models tend to give coherent post-hoc rationalizations for refusal, and that these are often divorced from the “real” underlying cause of refusal. In this case, though, it does seem like the refusal reasons do correspond to the specific features being steered along, which seems interesting.
Seems right, nice!