By several reports, (e.g. here and here) OpenAI is throwing enormous amounts of training compute at o-series models. And if the new RL paradigm involves more decentralized training compute than the pretraining paradigm, that could lead to more consolidation into a few players, not less, because pretraining* is bottlenecked by the size of the largest cluster. E.g. OpenAI’s biggest single compute cluster is similar in size to xAI’s, even though OpenAI has access to much more compute overall. But if it’s just about who has the most compute then the biggest players will win.
*though pretraining will probably shift to distributed training eventually
By several reports, (e.g. here and here) OpenAI is throwing enormous amounts of training compute at o-series models. And if the new RL paradigm involves more decentralized training compute than the pretraining paradigm, that could lead to more consolidation into a few players, not less, because pretraining* is bottlenecked by the size of the largest cluster. E.g. OpenAI’s biggest single compute cluster is similar in size to xAI’s, even though OpenAI has access to much more compute overall. But if it’s just about who has the most compute then the biggest players will win.
*though pretraining will probably shift to distributed training eventually