data analyst at Epoch AI
@justjoshinyou13 on twitter
Josh You
Karma: 228
The RE-bench result is just for five tasks, the second graph is for a broader task suite of almost 200 tasks. I wouldn’t read much into o3 doing worse than other models at RE-bench because of the small sample.
One factor is how compute-constrained capabilities research is compared to safety research.
There’s been lots of ink spilled recently on the degree to which capabilities research is compute-bottlenecked in a scenario where you have a surge of AI research labor caused by automated AI researchers, but everyone agrees that capability research requires a ton of compute. For safety research, I don’t feel well informed about how important compute is, but I think a lot of safety research being done right now is not very compute-intensive. I’m not aware off the top of my head of safety work that involve doing experiments approaching the scale of training a frontier model. So you could have a scenario where the large majority of resources is going towards capabilities, but the big increase in safety labor in absolute terms differentially helps safety.
Other examples:
In 1914, Germany had a population of 67 million, about 70% of the US’s population of 98 million. In 1939, Germany’s population was still over half of the US’s. Today Germany has about 25% as many people.
The Soviet Union had a higher population than the US throughout the Cold War. Russia, which only contains about half of the population of the former Soviet Union, has about 42% of the US population today.
But this only works if those less worried about AI risks who join such a collaboration don’t use the knowledge they gain to cash in on the AI boom in an acceleratory way. Doing so undermines the very point of such a project, namely, to try to make AI go well. It is incredibly damaging to trust within the community.
...This is less about attacking those three folks and more just noting that we need to strive to avoid situations where things like this happen in the first place.
(note: I work at Epoch) This attitude feels like a recipe for creating an intellectual bubble. Of course people will use the knowledge they gain in collaboration with you for the purposes that they think are best. I think it would be pretty bad for the AI safety community if it just relied on forecasting work from card-carrying AI safety advocates.
I think there are two models that you measured time horizon for, Claude 3 Opus, and GPT-4 Turbo, that didn’t make it onto the main figure. Is that right? There are 13 models in Figure 5, which shows the time horizon curves for a bunch of models across the full test suite, and only 11 dots on Figure 1.
AI has probably increased valuations for Big Tech (particularly Nvidia) by at least a few trillion over the past two years. So part of this is that investors think OpenAI/Anthropic will only capture around 10% of total AI profits.
65T tokens doesn’t get you to 1e26 FLOP with 100B active params? You’d need well over 100T tokens: 6 * 100 billion * 65 trillion is 3.9e25 FLOP.
GPT-4.5 being trained on fewer tokens than GPT-4o doesn’t really make sense. GPT-4.5 only having 5x more active params than GPT-4o doesn’t quite make sense either, though I’m not as confident that’s wrong.
1e26 FLOP would have had a significant opportunity cost. Remember that OpenAI was and is very GPU constrained and may have valued GPU hours in a large-scale cluster a lot more than $2/hour. It would be worth it to make your flagship model good, but not worth it if it barely has any effect on your flagship model. I don’t think it’s a good idea to reason backwards from alleging some compute budget that OpenAI might have had at X date, to inferring the training FLOP of a model trained then.
I don’t think GPT-4o was trained on 1e26 FLOP or particularly close to it. Overtraining is common but GPT-4o being overtrained by 10x for 1e26 FLOP is kind of a strong and surprising claim (some models like Llama 3 8b are extremely overtrained but they’re small so this overtraining is cheap). I think a more natural explanation is that it improves on GPT-4 because of superior post-training and other innovations.
The high cost and slow speed of GPT-4.5 seems like a sign OpenAI is facing data constraints, though we don’t actually know the parameters and OpenAI might be charging an bigger margin than usual (it’s a “research preview” not a flagship commercial product). If data was more abundant, wouldn’t GPT-4.5 be more overtrained and have fewer parameters?
edit: FWIW Artificial Analysis measures GPT-4.5 at a not-that-bad 50 tokens per second whereas I’ve been experiencing a painfully slow 10-20 tokens/second in the chat app. So may just be growing pains until they get more inference GPUs online. But OpenAI does call it a “chonky” model, implying significant parameter scaling.
if OpenAI follows the usual naming convention of roughly 100x in raw compute.
I doubt this is a real convention. I think OpenAI wanted to call Orion GPT-5 if they thought it was good enough to deserve the name.
In Holden Karnofsky’s “AI Could Defeat All Of Us Combined” a plausible existential risk threat model is described, in which a swarm of human-level AIs outmanoeuvre humans due to AI’s faster cognitive speeds and improved coordination, rather than qualitative superintelligence capabilities. This scenario is predicated on the belief that “once the first human-level AI system is created, whoever created it could use the same computing power it took to create it in order to run several hundred million copies for about a year each.” If the first AGIs are as expensive to run as o3-high (costing ~$3k/task), this threat model seems much less plausible.
I wonder how different the reasoning paradigm is, actually, from the picture presented here. After all, running a huge number of AI copies in parallel is… scaling up test-time compute.
The overhang argument is a rough analogy anyway. I think you are invoking the intuition of replacing the AI equivalent of a very large group of typical humans with the AI equivalent of a small number of ponderous geniuses, but those analogies are going to be highly imperfect in practice.
By several reports, (e.g. here and here) OpenAI is throwing enormous amounts of training compute at o-series models. And if the new RL paradigm involves more decentralized training compute than the pretraining paradigm, that could lead to more consolidation into a few players, not less, because pretraining* is bottlenecked by the size of the largest cluster. E.g. OpenAI’s biggest single compute cluster is similar in size to xAI’s, even though OpenAI has access to much more compute overall. But if it’s just about who has the most compute then the biggest players will win.
*though pretraining will probably shift to distributed training eventually
AI systems can presumably be given at least as much access to company data as human employees at that company. So if rapidly scaling up the number and quality of human workers at a given company would be transformative, AI agents with >=human-level intelligence can also be transformative.
I think a little more explanation is required on why there isn’t already a model with 5-10x* more compute than GPT-4 (which would be “4.5 level” given that GPT version numbers have historically gone up by 1 for every two OOMs, though I think the model literally called GPT-5 will only be a roughly 10x scale-up).
You’d need around 100,000 H100s (or maybe somewhat fewer; Llama 3.1 was 2x GPT-4 and trained using 16,000 H100s) to train a model at 10x GPT-4. This has been available to the biggest hyperscalers since sometime last year. Naively it might take ~9 months from taking delivery of chips to releasing a model (perhaps 3 months to set up the cluster, 3 months for pre-training, 3 months of post-training, evaluations, etc). But most likely the engineering challenges in building a cluster that big, which is unprecedented, and perhaps high demand for inference, has prevented them from concentrating that much compute into one training run in time to release a model by now.
*I’m not totally sure the 5x threshold (1e26 FLOP) hasn’t been breached but most people think it hasn’t.
Llama 405B was trained on a bunch of synthetic data in post-training for coding, long-context prompts, and tool use (see section 4.3 of the paper).
AI that can rewrite CUDA is a ways off. It’s possible that it won’t be that far away in calendar time, but it is far away in terms of AI market growth and hype cycles. If GPT-5 does well, Nvidia will reap the gains more than AMD or Google.
The US is currently donating doses to other countries in large quantities. Domestically, it has around 54m doses distributed but not used right now. (https://covid.cdc.gov/covid-data-tracker/#vaccinations). Some but certainly not all of those are at risk of expiration. If US authorities recommended booster shots for the general population then that would easily use up the currently unused supply and reduce vaccine exports.
I did it, I did it, I did it, yay!
A compromise that I find appealing and might implement for myself is giving a fixed percentage over a fixed amount, with that fixed percentage being relatively high (well above ten percent). You could also have multiple “donation brackets” with an increased marginal donation rate as your income increases.
I don’t understand Hinge #2. Wall clock time for equally large training runs 18 months apart could easily shrink by 3⁄4 for banal reasons like larger training clusters. why would this be evidence of software-only acceleration?