What is troubling is that so many others (now at least Anthropic, Google, Meta and xAI, potentially a few others too) matched 4-level without getting above about a 4.2.
Last year the same impression held about getting above ChatGPT-3.5. The “widely available” thing right now seems to be the 20K H100s clusters, which means 5e25-1e26 FLOPs in a training run that lasts 2-3 months. And so everyone has their own GPT-4, which proves that there is no essential secret sauce at this level of capabilities, only scale. Going further awaits completion of larger training datacenters, with Anthropic and OpenAIprobablyalready training at the next level of scale. Hyperscalersaresinking about $50bn a year each in capex, so $1bn training runs are not yet something outrageous.
Last year the same impression held about getting above ChatGPT-3.5. The “widely available” thing right now seems to be the 20K H100s clusters, which means 5e25-1e26 FLOPs in a training run that lasts 2-3 months. And so everyone has their own GPT-4, which proves that there is no essential secret sauce at this level of capabilities, only scale. Going further awaits completion of larger training datacenters, with Anthropic and OpenAI probably already training at the next level of scale. Hyperscalers are sinking about $50bn a year each in capex, so $1bn training runs are not yet something outrageous.