Rohin Shah comments on Probability that other architectures will scale as well as Transformers?

[Rohin Shah]

It sounds like you are saying the task is more important than the architecture, so we should talk less about architectures and more about tasks.

Yes, that’s right.

For example, when thinking about the safety or generalization properties of a system the architecture might be more important, no?

I’d be pretty surprised if this were the case after conditioning on the raw capabilities of the architecture, though I can’t rule it out.