I was thinking in similar lines, but eventually dropped it because I felt like the gradients would likely miss something if e.g. a saturated softmax prevents any gradient from going through. I find it interesting that experiments also find that the interaction basis didn’t work, and I wonder whether any of the failure here is due to saturated softmaxes.
I doubt it. Evaluating gradients along an entire trajectory from a baseline gave qualitatively similar results.
A saturated softmax also really does induce insensitivity to small changes. If two nodes are always connected by a saturated softmax, they can’t be exchanging more than one bit of information. Though the importance of that bit can be large.
My best guess for why the Interaction Basis didn’t work is that sparse, overcomplete representations really are a thing. So in general, you’re not going to get a good decomposition of LMs from a Cartesian basis of activation space.
I was thinking in similar lines, but eventually dropped it because I felt like the gradients would likely miss something if e.g. a saturated softmax prevents any gradient from going through. I find it interesting that experiments also find that the interaction basis didn’t work, and I wonder whether any of the failure here is due to saturated softmaxes.
I doubt it. Evaluating gradients along an entire trajectory from a baseline gave qualitatively similar results.
A saturated softmax also really does induce insensitivity to small changes. If two nodes are always connected by a saturated softmax, they can’t be exchanging more than one bit of information. Though the importance of that bit can be large.
My best guess for why the Interaction Basis didn’t work is that sparse, overcomplete representations really are a thing. So in general, you’re not going to get a good decomposition of LMs from a Cartesian basis of activation space.