It seems the 100K H100s for the Memphis datacenter can plausibly get online around the end of 2024, and planned release of Grok-3 gives additional indirect evidence this might be the case. While OpenAI might have started training in May on a cluster that might have 100K H100s as well. So I’m updating my previous guess of xAI being 18 months behind to them only being 7-9 months behind for the 100K H100s scale (above 4e26 FLOPs).
It seems the 100K H100s for the Memphis datacenter can plausibly get online around the end of 2024, and planned release of Grok-3 gives additional indirect evidence this might be the case. While OpenAI might have started training in May on a cluster that might have 100K H100s as well. So I’m updating my previous guess of xAI being 18 months behind to them only being 7-9 months behind for the 100K H100s scale (above 4e26 FLOPs).