I with more of the language alignment research folks were looking into how current proposals for aligning transformers end up working on S4 models.
(I am one of said folks so maybe hypocritical to not work on it)
In particular it seems like there’s way in which it would be more interpretable than transformers:
adjustable timescale stepping (either sub-stepping, or super-stepping time)
approximately separable state spaces/dynamics—this one is crazy conjecture—it seems like it should be possible to force the state space and dynamics into separate groups, in ways that would allow analysis of them in isolation or in relation to the rest of the model
It does seem like they’re not likely to be competitive with transformers for short-context modeling anytime soon, but if they end up being differentially alignment-friendly, then we could instead try to make them more competitive.
(In general I think it’s much easier to make an approach more competitive than it is to make it more aligned)
I with more of the language alignment research folks were looking into how current proposals for aligning transformers end up working on S4 models.
(I am one of said folks so maybe hypocritical to not work on it)
In particular it seems like there’s way in which it would be more interpretable than transformers:
adjustable timescale stepping (either sub-stepping, or super-stepping time)
approximately separable state spaces/dynamics—this one is crazy conjecture—it seems like it should be possible to force the state space and dynamics into separate groups, in ways that would allow analysis of them in isolation or in relation to the rest of the model
It does seem like they’re not likely to be competitive with transformers for short-context modeling anytime soon, but if they end up being differentially alignment-friendly, then we could instead try to make them more competitive.
(In general I think it’s much easier to make an approach more competitive than it is to make it more aligned)