Situational Awareness Summarized—Part 2

This is the second post in the Situational Awareness Summarized sequence. Collectively, these posts represent my attempt to condense Leopold Aschenbrenner’s recent report, Situational Awareness, into something more digestible.

Part II: Zoom to Foom

This section outlines how the development of AGI quickly and inevitably leads to superintelligence.

Leopold points out that a massive increase in AI research progress doesn’t require that we figure out how to automate everything—just that we figure out how to automate AI research. Building on the “drop-in remote worker” concept from Part 1, Leopold argues that this is plausible—and that it scales. With a few more years of growth in our compute capacity, we’d be able to run as many as 100 million “human-equivalent” researchers, and possibly speed them up as well. Part II predicts that automated researchers could give us a decade’s progress (5+ OOMs) in a single year. If this happens, it likely translates to extremely rapid progress in AI capabilities—and, shortly thereafter, in everything else.

Supporting this case, Leopold outlines some of the advantages that automated researchers might have over human researchers. In particular, he highlights the increased memory and competency of AGI-level thinkers, the ability to copy and replicate after “onboarding”, and the ability to rapidly turn new machine learning efficiencies into faster and better thinking (since they run on the same software they’re researching).

Potential Bottlenecks

He also describes some potential bottlenecks to this process, and how they might be overcome. Possible bottlenecks include:

  • Limited compute available for experiments

  • Complementarities and long tails—the hardest fraction of work will govern speed

  • Fundamental limits to algorithmic efficiencies

  • Other sources of diminishing returns to research

Compute for experiments

Leopold considers this the most important bottleneck. Pure cognitive labor alone can’t make rapid progress; you also need to run experiments. Some of these experiments will take vast quantities of compute. Leopold suggests some ways that automated researchers might work around this problem:

  • Running small tests

  • Running a few large tests with huge gains

  • Increasing effective compute by compounding efficiencies

  • Spending lots of researcher-time to avoid bugs

  • Having better intuitions overall (by being smarter and reading all prior work)

In an aside, Leopold also addresses a possible counterargument: why haven’t the large number of academic researchers increased progress at labs? He argues that automated researchers would be very different from academics, and in particular would have access to the newest models being actively developed.

Complementarities and long tails

A valuable lesson from economics is that it is often difficult to fully automate anything, because many kinds of work are complementary, and the work that is hardest to automate becomes the new bottleneck. Leopold broadly accepts this model, and believes that it will slow down the automated research pipeline by a few years, but no more than that.

Leopold also points out that progress in capabilities may be uneven. Models might be superhuman at coding before they are human-level at research. This suggests we would see a sudden jump once models passed the last few hurdles to full automation, since they would already be superhuman in some domains.

Fundamental limits to algorithmic efficiencies

Models of the same size won’t keep getting smarter forever; eventually we will run out of new techniques like chain-of-thought prompting that improve performance of a given model size. Will this slow progress? Leopold thinks it will eventually, but argues that there is probably still low-hanging fruit to pluck. He estimates another 5 OOMs of efficiency might be achievable in the next decade (and that this might be achieved much faster by automated researchers).

Diminishing returns

Technological progress tends to slow over time, as new ideas get harder to find. How will this affect AI research in the next decade? Leopold makes the case that the sudden jump in researcher output from automation will (temporarily) overcome the usual growth-slowing effect of diminishing returns. In particular, he argues that it would be extremely unlikely for ideas to suddenly get much harder to find at exactly the same rate as automation unlocks much faster research. His bottom line: we’ll run out of good ideas eventually, but we’ll probably get to superintelligence first.

Implications of Superintelligence

Leopold argues that the minds output by the above process will be quantitatively and qualitatively superhuman. Billions of AIs that read and think much faster than humans, able to come up with ideas that no human has ever considered, will rapidly eclipse human capabilities. Leopold predicts these minds will:

  • Automate all cognitive work,

  • Solve robotics (which Leopold considers mostly a ML algorithms problem today),

  • Rapidly develop new technologies,

  • Cause an explosion of industrial and economic growth,

  • Provide an overwhelming military advantage, and

  • Become capable of overthrowing major governments

Based on the advancements above, Leopold predicts a volatile and dangerous period in the coming decade during which humanity may lose control of our own future.

Some questions I have after reading

  • I still feel like “a decade of progress in one year” is an oddly specific prediction. Why not a century? A millenium? With this report throwing out orders of magnitude like popcorn, it seems weird that “100 million fast polymath automated researchers running at 10x speed and improving their own cognition” would just happen to hit enough bottlenecks to only accelerate progress by a factor of 10. But the basic idea of rapid progress still makes sense.

  • I wonder how effective it would be to make so many duplicate researchers. Between communication costs, error-correcting, and duplicate work, 100 million AIs might run into a Brooks’s Law problem and be less effective than we might naively think. (But again, we’re probably still looking at very rapid progress).

  • Why would complementarities only delay the start of this accelerating process by a few years? I didn’t really see a great argument for that specific timeline in the report. It seemed like more of a gut-level guess. I would love to hear Leopold’s probability that this kicks off in e.g. 2029 instead.

  • Employing what is effectively a slave population of 100 million super smart researchers seems like a very unstable position. Leopold devotes a single paragraph to the possibility that these AIs might simply take over—a feat he himself argues they could quite easily accomplish—and nothing at all to the prospect of safely or ethically preventing this outcome. I expect to read more about this in the Superalignment section, but it still seems to me that this section is making a huge assumption. Why would 100 million AGIs listen to us in the first place?