So, it’s not clear that they got the target performance out of this model. However, they did manage to scale it, which is all it takes. They don’t need to buy more GPUs, they’ve got what they need, as long as they can find the algorithms. Which are mostly published.
Well they get to run however much compute they do have 6 more months with no competition. Probably several years since obviously this pause would get renewed again and again until someone honoring it defects. Note that enormous models are a function of total cluster memory and interconnect. Many current clusters have enough memory for theoretically enormous models, 10 trillion weights plus. Having too few GPUs so training takes a year+ is a problem unless your competition is all idle.
So, it’s not clear that they got the target performance out of this model. However, they did manage to scale it, which is all it takes. They don’t need to buy more GPUs, they’ve got what they need, as long as they can find the algorithms. Which are mostly published.
https://twitter.com/arankomatsuzaki/status/1637983258880122881 - https://arxiv.org/abs/2303.10845
Thanks! Haven’t found good comments on that paper (and lack the technical insights to evaluate it myself)
Are you implying that China has access to compute required for a) GPT-4 type models or b) AGI?
Well they get to run however much compute they do have 6 more months with no competition. Probably several years since obviously this pause would get renewed again and again until someone honoring it defects. Note that enormous models are a function of total cluster memory and interconnect. Many current clusters have enough memory for theoretically enormous models, 10 trillion weights plus. Having too few GPUs so training takes a year+ is a problem unless your competition is all idle.
a.