The fact that RL seems to be working well on LLMs now, without special tricks, as reported by many replications of r1, suggests to me that AGI is indeed not far off.
Still, at least as long as base model effective training compute isn’t scaled another 1,000x (which is 2028-2029), this kind of RL training probably won’t generalize far enough without neural (LLM) rewards, which for now don’t let RL scale as much as with explicitly coded verifiers.
This is an obvious thing to try, but it’s not what currently already works, and it’s not certain to work without some additional ideas. You can do a little bit of this, but not nearly to the extent that o1/R1 inch towards saturating benchmarks on math/coding olympiad-like problems. So long as using LLMs as reward for scalable RL doesn’t work yet, supercharged capabilities of o1/R1-like models plausibly remain restricted to verifiable tasks.
Relative to GPT-4o, which was trained at a time when 30K H100s clusters were around, and so in BF16 could be expected to be around 8e25 FLOPs, possibly overtrained to a degree that’s not too different from DeepSeek-V3 itself.
Amodei’s post you linked says a few of tens of millions of dollars for Claude 3.5 Sonnet, which is maybe 4e25 FLOPs in BF16, but I think Claude 3.5 Sonnet is better than DeepSeek-V3, which is not as clearly the case for GPT-4o and DeepSeek-V3, making them easier to compare. Being better than GPT-4o at 2x fewer FLOPs, Claude 3.5 Sonnet has at least a 4x compute multiplier over it (under all the assumptions), but not necessarily more, while with DeepSeek-V3 there’s evidence for more. As DeepSeek-V3 was trained more than half a year later than Claude 3.5 Sonnet, it’s somewhat “on trend” in getting a compute multiplier of 16x instead of 4x, if we anchor to Amodei’s claim of 4x per year.
Stargate is evidence towards slower training system scaling. The rumored reason for starting the project is that Microsoft isn’t building giant frontier training systems fast enough, probably because they aren’t seeing the case for doing that faster. In which case other hyperscalers might think similarly, and they are the most well-positioned to build these systems, so this attitude might be indicative of how frontier training systems get built overall, which is notably slower than technically feasible.
The $80bn Microsoft capex is not relevant to this if it goes to many smaller systems[1], which is only natural as there are millions of datacenter GPUs but only a few 100K GPU frontier training systems, a tiny fraction of inference and smaller/research training compute. The $500bn figure is not relevant as for now it’s only a vague plan. But Microsoft not agreeing to build training systems on OpenAI’s schedule is some evidence.
OpenAI would want to get from under Microsoft’s thumb anyway[2], and this gets ever more difficult over time, since frontier training systems get ever more expensive, so the sooner they try the more likely they are to succeed. But even this consideration is some evidence of slowdown, since it only motivates saying you want to build frontier training systems even faster, but doesn’t in itself motivate actually going through with it, beyond building a competitive training system that makes you independent.
So the clues that support the prospect of scaling to 1 GW in 2025 and to 5 GW in 2027 could be misleading, running contrary to hyperscaler attitudes and not aligning even with OpenAI’s immediate incentives.
I previously expected that $80bn is evidence that they are building a large training system this year, but it now seems that they are building more inference instead.
As Satya Nadella said, “If OpenAI disappeared tomorrow… we have all the IP rights and all the capability. We have the people, we have the compute, we have the data, we have everything. We are below them, above them, around them.”
From what I remember, the training-compute optimal number of experts was like 64
I think it only gets better with more experts if you keep the number of active parameters unchanged. Is there some setting where it gets worse after a while? There certainly are engineering difficulties and diminishing returns.
Also, the number of activated experts can vary (there are 8 activated routed experts in DeepSeek-V3 out of the total of 256), so “number of experts” doesn’t really capture the ratio of total to activated, probably not a good anchor by itself.
Given newer implementations and aiming for inference-compute optimality, it seems logical that more than 64 experts could be great.
This still doesn’t help with the question of why 37B active parameters is sufficient. Even with 100500 experts you can’t expect 1B active parameters to be sufficient to maintain GPT-4 quality. The rumor for original GPT-4 is that it has 2 activated experts out of 16 in total, so the ratio is 1:8, while for DeepSeek-V3 it’s 1:32.
that’s why I wrote: “possibly 4x fewer training steps for the same number of tokens if predicting tokens only once” (assuming predicting 4 tokens at a time), but that’s not demonstrated nor published (given my limited knowledge on this)
Not sure how to parse this, my point is that the number of training steps remains the same, training efficiency doesn’t significantly increase, there’s even slight overhead from adding the predict-the-token-after-next blocks of parameters. This is described in Section 2.2 of the paper. You get better speculative decoding at inference time (and also better quality of output), but training time is the same, not with 2x fewer or 4x fewer steps.
32B active parameters instead of likely ~
220280B for GPT4 ⇒6.88.7x lower training cost per token.
It’s 37B active parameters, not 32B.
The bet that “makes sense” is that quality of Claude 3.6 Sonnet, GPT-4o and DeepSeek-V3 is the best that we’re going to get in the next 2-3 years, and DeepSeek-V3 gets it much cheaper (less active parameters, smaller margins from open weights), also “suggesting” that quality is compute-insensitive in a large range, so there is no benefit from more compute per token.
But if quality instead improves soon (including by training DeepSeek-V3 architecture on GPT-4o compute), and that improvement either makes it necessary to use more compute per token, or motivates using inference for more tokens even with models that have the same active parameter count (as in Jevons paradox), that argument doesn’t work. Also, the ceiling of quality at the possible scaling slowdown point depends on efficiency of training (compute multiplier) applied to the largest training system that the AI economics will support (maybe 5-15 GW without almost-AGI), and improved efficiency of DeepSeek-V3 raises that ceiling.
Taken in isolation, DeepSeek-V3 looks like a 15x compute multiplier. But if a lot of it is data, the multiplier won’t scale (when you need much more data, it necessarily becomes worse, or alternatively you need a teacher model that’s already better). In any case, this raises the ceiling for what 5 GW training systems can do (at which point there’s either almost-AGI or scaling slows down a lot). And there the 15x multiplier of DeepSeek-V3 (or what remains of it after scaling) needs to be compared with the algorithmic advancements of 2025-2028, which would’ve included most of the things in DeepSeek-V3 anyway, so the counterfactual impact is small.
32B active parameters instead of likely ~220B for GPT4
It’s 37B instead of maybe 280B (non-expert parameters also count), but in any case the question is how this manages to maintain quality. If this wasn’t an issue, why not 8B active parameters, or 1M active parameters?
32B active parameters instead of likely ~220B for GPT4 ⇒ 6.8x lower training … cost
Doesn’t follow, training cost scales with the number of training tokens. In this case DeepSeek-V3 uses maybe 1.5x-2x more tokens than original GPT-4.
The training costs are maybe 5e24 FLOPs and 2e25 FLOPs, differ by 4x. DeepSeek-V3 is better than original GPT-4 though, you need to compare with GPT-4o, which almost certainly uses more compute in training than original GPT-4 (maybe 4x more, so maybe 16x more than DeepSeek-V3).
8bits training instead of 16bits ⇒ 4x lower training cost
FLOP/s for FP8 are almost always 2x the FLOP/s for BF16, not 4x.
Multi-token training ⇒ ~2x training efficiency
You still train on every token. There is an additional “layer” in model parameters that predicts the token-after-next (Figure 3 in the paper), so there’s a bit of overhead in training (not much, with 61 total layers). The results are better, but not that much better (Table 4).
training on O1 outputs
Outputs of o1 don’t include reasoning traces, so not particularly useful compared to outputs of chatbot models, and very expensive, so only a modest amount can be collected.
Imitation helps with post-training, but the compute-heavy part is pretraining, and obtaining good quality with little pretraining is a novel feat that isn’t known to be explainable by good post-training, or by including a lot of outputs from good models in the pretraining/annealing mix.
This seems unlikely to be a neglected concern, unless there are specific signs that it is.
could end up being the most important thing I’ve ever written
The $6 million is disputed by a video arguing that DeepSeek used far more compute than they admit to.
The prior reference is a Dylan Patel tweet from Nov 2024, in the wake of R1-Lite-Preview release:
Deepseek has over 50k Hopper GPUs to be clear.
People need to stop acting like they only have that 10k A100 cluster.
They are omega cracked on ML research and infra management but they aren’t doing it with that many fewer GPUs
DeepSeek explicitly states that
DeepSeek-V3 requires only 2.788M H800 GPU hours for its full training.
This seems unlikely to be a lie, the reputational damage would’ve motivated not mentioning amount of compute instead, but the most interesting thing about DeepSeek-V3 is precisely this claim, that its quality is possible with so little compute.
Certainly designing the architecture, the data mix, and the training process that made it possible required much more compute than the final training run, so in total it cost much more to develop than $6 million. And the 50K H100/H800 system is one way to go about that, though renting a bunch of 512-GPU instances from various clouds probably would’ve sufficed as well.
Found the following in the Jan 23 newsletter:
AI doesn’t accelerate my writing much, although it is often helpful in parsing papers and helping me think through things. But it’s a huge multiplier on my coding, like more than 10x.
What actually happens with xAI and Anthropic compute by end of 2025 is less clear. For xAI, 300K B200s figure was mentioned in June 2024. For Anthropic, Amodei said in a recent interview that
I would not be surprised if in 2026 we have more than a million of some kind of chip.
Meanwhile, xAI will have a 200K H100/H200 system, and Anthropic a 400K Trn2 system, which is about 250K H100s worth of FLOP/s (ready by a few months into 2025). The 400-600 MW at Abilene site for OpenAI are 200K-300K B200s, which is about 500K-750K H100s worth of FLOP/s.
Google might start 2026 with the largest training system among the big labs, by a factor of about 2x, at about 1 GW.
OpenAI/Microsoft Stargate schism suggests that compute being built this year by Microsoft is unlikely to form part of a geographically distributed training system that also includes compute being built at Abilene site. Seems like OpenAI will be building its own training systems (through Stargate), while Microsoft will be serving inference (and possibly generation for RL training, but it remains unclear if it can be an important fraction of pretraining budget in 2025-2026). Thus only 400-600 MW of GB200s by end of 2025 for an OpenAI training system, not 1 GW.
Meta announced a 2 GW datacenter at Richland Parish site, but 1 GW for 2025 seems to be across all datacenters, not for a single training system. So the training system will be smaller by end of 2025.
By “3rd person perspective” I mean considering the world itself, there is no actual third person needed for it. It’s the same framing as used by a physicist when talking about the early stages of the universe when humans were not yet around, or when talking about a universe with alternative laws of physics, or when talking about a small system that doesn’t include any humans as its part. Or when a mathematician talks about a curve on a plane.
Knowing absolutely everything is not necessary to know the relevant things, and in this case we know all the people at all times, and the states of their minds, their remembered experiences and possible reasoning they might perform based on those experiences. Observations take time and cognition to process, they should always be considered from slightly in the future relative to when raw data enters a mind. So it’s misleading to talk about a person that will experience an observation shortly, and what that experience entails, the clearer situation is looking at a person who has already experienced that observation a bit in the past and can now think about it. When a copied person looks back at their memories, or a person about to be copied considers what’s about to happen, the “experience” of being copied is nowhere to be found, there is only the observation of the new situation that the future copies find themselves in, and that has nothing to do with the splitting into multiple copies of the person from the past.
How did DeepSeek accidentally happen to invest precisely the amount of compute into V3 and r1 that would get them into the capability region of GPT-4/o1
Selection effect. If DeepSeek-V2.5 was this good, we would be talking about it instead.
GPT-4 was supposedly trained for $100 million, and V3 for $5.5 million
Original GPT-4 is 2e25 FLOPs and compute optimal, V3 is about 5e24 FLOPs and overtrained (400 tokens/parameter, about 10x-20x), so a compute optimal model with the same architecture would only need about 3e24 FLOPs of raw compute[1]. Original GPT-4 was trained in 2022 on A100s and needed a lot of them, while in 2024 it could be trained on 8K H100s in BF16. DeepSeek-V3 is trained in FP8, doubling the FLOP/s, so the FLOPs of original GPT-4 could be produced in FP8 by mere 4K H100s. DeepSeek-V3 was trained on 2K H800s, whose performance is about that of 1.5K H100s. So the cost only has to differ by about 3x, not 20x, when comparing a compute optimal variant of DeepSeek-V3 with original GPT-4, using the same hardware and training with the same floating point precision.
The relevant comparison is with GPT-4o though, not original GPT-4. Since GPT-4o was trained in late 2023 or early 2024, there were 30K H100s clusters around, which makes 8e25 FLOPs of raw compute plausible (assuming it’s in BF16). It might be overtrained, so make that 4e25 FLOPs for a compute optimal model with the same architecture. Thus when comparing architectures alone, GPT-4o probably uses about 15x more compute than DeepSeek-V3.
toy model … f(x) = Ax and g(x) = Bx, where x is the compute invested
Returns on compute are logarithmic though, advantage of a $150 billion training system over a $150 million one is merely twice that of $150 billion over $5 billion or $5 billion over $150 million. Restrictions on access to compute can only be overcome with 30x compute multipliers, and at least DeepSeek-V3 is going to be reproduced using the big compute of US training systems shortly, so that advantage is already gone.
That is, raw utilized compute. I’m assuming the same compute utilization for all models.
all copies … will claim to be the original … regardless of whether they are the original
Not if they endorse Litany of Tarski and understand the thought experiment!
Any “perceive yourself to X” phenomenon is something that happens within cognition of some abstract agent/person instance, whether they exist in some world or not. What kind of person instance is “perceiving themselves to black out” (that is, having blacked out)? Ghosts and afterlife seem more grounded than that. But for Earth/Mars question, both options are quite clear, and there is a you that perceives either of them in some of the possibilities, we can point to where those that perceive each of them are, and that is what would be correct for those instances to conclude about themselves, that they exist in the situations that contain them, known from the statement of the thought experiment.
Not to convergence, the graphs in the paper keep going up. Which across the analogy might explain some of the change from o1 to o3 (the graphs in the o1 post also keep going up), though new graders coded for additional verifiable problems are no doubt a large part of it as well.
It seems like o1-mini is its own thing, might even start with a base model that’s unrelated to GPT-4o-mini (it might be using its own specialized pretraining data mix). So a clue about o3-mini data doesn’t obviously transfer to o3.
The numbering in GPT-N series advances with roughly 100x in raw compute at a time. If original GPT-4 is 2e25 FLOPs, then a GPT-5 would need 2e27 FLOPs, and a 100K H100s training system (like the Microsoft/OpenAI system at the site near the Goodyear airport) can only get you 3e26 FLOPs or so (in BF16 in 3 months). The initial Stargate training system at Abilene site, after it gets 300K B200s, will be 7x stronger than that, so will be able to get 2e27 FLOPs. Thus I expect GPT-5 in 2026 if OpenAI keeps following the naming convention, while the new 100K H100s model this year will be GPT-4.5o or something like that.