Zach Furman comments on Is the speed of training large models going to increase significantly in the near future due to Cerebras Andromeda?

[Zach Furman]

Hmm, I see how that would happen with other architectures, but I’m a bit confused how this is $O\left(n^2\right)$ here? Andromeda has the weight updates computed by a single server (MemoryX) and then distributed to all the nodes. Wouldn’t this be a one-to-many broadcast with $O\left(n\right)$ transmission time?

[Razied]

You’re completely right, I don’t know how I missed that, I must be more tired than I thought I was.