Tom Davidson

Will com­pute bot­tle­necks pre­vent a soft­ware in­tel­li­gence ex­plo­sion?

• Tom DavidsonMar 28, 2025, 6:19 PM

  2 points

  0

  in reply to: Ram Potham’s comment on: Will AI R&D Au­toma­tion Cause a Soft­ware In­tel­li­gence Ex­plo­sion?

  Agreed there’s an ultimate cap on software improvements—the worry is that it’s very far away!

• Tom DavidsonMar 28, 2025, 6:17 PM

  8 points

  0

  in reply to: PeterMcCluskey’s comment on: Three Types of In­tel­li­gence Explosion

  It does sound like a lot—that’s 5 OOMs to reach human learning efficiency and then 8 OOMs more. But when we BOTECed the sources of algorithmic efficiency gain on top of the human brain, it seemed like you could easily get more than 8. But agreed it seems like a lot. Though we are talking about ultimate physical limits here!

  Interesting re the early years. So you’d accept that learning from ⁵⁄₆ could be OOMs more efficient, but would deny that the early years could be improved?

  Though you’re not really speaking to the ‘undertrained’ point, which is about the number of params vs data points

• Tom DavidsonMar 28, 2025, 6:13 PM

  1 point

  0

  in reply to: ryan_greenblatt’s comment on: Three Types of In­tel­li­gence Explosion

  I expect that full stack intelligence explosion could look more like “make the whole economy bigger using a bunch of AI labor” rather than specifically automating the chip production process. (That said, in practice I expect explicit focused automation of chip production to be an important part of the picture, probably the majority of the acceleration effect.) Minimally, you need to scale up energy at some point.

  Agreed on the substance, we just didn’t explain this well.
  

  • You talk about “chip technology” feedback loop as taking months, but presumably improvements to ASML take longer as they often require building new fabs?

  Agreed!

  Re Flop/​joule also agree on the substance—we went with FLOP/​joule bc we wanted a clean estimate for the OOMs before reaching limits for each factor. I believe our estimate of the total OOMs to limits (including both chip tech and chip production) is right, but you’re right that there are ways to intutively improve chip tech that don’t increase FLOP/​joule
  
  

Will the Need to Re­train AI Models from Scratch Block a Soft­ware In­tel­li­gence Ex­plo­sion?

Will AI R&D Au­toma­tion Cause a Soft­ware In­tel­li­gence Ex­plo­sion?

Three Types of In­tel­li­gence Explosion

Forethought: a new AI macros­trat­egy group

Could Ad­vanced AI Ac­cel­er­ate the Pace of AI Progress? In­ter­views with AI Researchers

• Tom DavidsonJan 30, 2025, 4:12 PM

  6 points

  0

  in reply to: Bogdan Ionut Cirstea’s comment on: Hu­man takeover might be worse than AI takeover

  I think rushing full steam ahead with AI increases human takeover risk

• Tom DavidsonJan 30, 2025, 4:07 PM

  1 point

  0

  in reply to: deep’s comment on: ryan_green­blatt’s Shortform

  Sure! See here: https://​​docs.google.com/​​document/​​d/​​1DZy1qgSal2xwDRR0wOPBroYE_RDV1_2vvhwVz4dxCVc/​​edit?tab=t.0#bookmark=id.eqgufka8idwl

• Tom DavidsonJan 22, 2025, 4:36 PM

  LW: 12 AF: 9

  0

  AF

  in reply to: ryan_greenblatt’s comment on: ryan_green­blatt’s Shortform

  Here’s my own estimate for this parameter:

  Once AI has automated AI R&D, will software progress become faster or slower over time? This depends on the extent to which software improvements get harder to find as software improves – the steepness of the diminishing returns.

  We can ask the following crucial empirical question:
  
  When (cumulative) cognitive research inputs double, how many times does software double?

  (In growth models of a software intelligence explosion, the answer to this empirical question is a parameter called r.)

  If the answer is “< 1”, then software progress will slow down over time. If the answer is “1”, software progress will remain at the same exponential rate. If the answer is “>1”, software progress will speed up over time.

  The bolded question can be studied empirically, by looking at how many times software has doubled each time the human researcher population has doubled.

  (What does it mean for “software” to double? A simple way of thinking about this is that software doubles when you can run twice as many copies of your AI with the same compute. But software improvements don’t just improve runtime efficiency: they also improve capabilities. To incorporate these improvements, we’ll ultimately need to make some speculative assumptions about how to translate capability improvements into an equivalently-useful runtime efficiency improvement..)

  The best quality data on this question is Epoch’s analysis of computer vision training efficiency. They estimate r = ~1.4: every time the researcher population doubled, training efficiency doubled 1.4 times. (Epoch’s preliminary analysis indicates that the r value for LLMs would likely be somewhat higher.) We can use this as a starting point, and then make various adjustments:

  • Upwards for improving capabilities. Improving training efficiency improves capabilities, as you can train a model with more “effective compute”. To quantify this effect, imagine we use a 2X training efficiency gain to train a model with twice as much “effective compute”. How many times would that double “software”? (I.e., how many doublings of runtime efficiency would have the same effect?) There are various sources of evidence on how much capabilities improve every time training efficiency doubles: toy ML experiments suggest the answer is ~1.7; human productivity studies suggest the answer is ~2.5. I put more weight on the former, so I’ll estimate 2. This doubles my median estimate to r = ~2.8 (= 1.4 * 2).

  • Upwards for post-training enhancements. So far, we’ve only considered pre-training improvements. But post-training enhancements like fine-tuning, scaffolding, and prompting also improve capabilities (o1 was developed using such techniques!). It’s hard to say how large an increase we’ll get from post-training enhancements. These can allow faster thinking, which could be a big factor. But there might also be strong diminishing returns to post-training enhancements holding base models fixed. I’ll estimate a 1-2X increase, and adjust my median estimate to r = ~4 (2.8*1.45=4).

  • Downwards for less growth in compute for experiments. Today, rising compute means we can run increasing numbers of GPT-3-sized experiments each year. This helps drive software progress. But compute won’t be growing in our scenario. That might mean that returns to additional cognitive labour diminish more steeply. On the other hand, the most important experiments are ones that use similar amounts of compute to training a SOTA model. Rising compute hasn’t actually increased the number of these experiments we can run, as rising compute increases the training compute for SOTA models. And in any case, this doesn’t affect post-training enhancements. But this still reduces my median estimate down to r = ~3. (See Eth (forthcoming) for more discussion.)

  • Downwards for fixed scale of hardware. In recent years, the scale of hardware available to researchers has increased massively. Researchers could invent new algorithms that only work at the new hardware scales for which no one had previously tried to to develop algorithms. Researchers may have been plucking low-hanging fruit for each new scale of hardware. But in the software intelligence explosions I’m considering, this won’t be possible because the hardware scale will be fixed. OAI estimate ImageNet efficiency via a method that accounts for this (by focussing on a fixed capability level), and find a 16-month doubling time, as compared with Epoch’s 9-month doubling time. This reduces my estimate down to r = ~1.7 (3 * ⁹⁄₁₆).

  • Downwards for diminishing returns becoming steeper over time. In most fields, returns diminish more steeply than in software R&D. So perhaps software will tend to become more like the average field over time. To estimate the size of this effect, we can take our estimate that software is ~10 OOMs from physical limits (discussed below), and assume that for each OOM increase in software, r falls by a constant amount, reaching zero once physical limits are reached. If r = 1.7, then this implies that r reduces by 0.17 for each OOM. Epoch estimates that pre-training algorithmic improvements are growing by an OOM every ~2 years, which would imply a reduction in r of 1.02 (6*0.17) by 2030. But when we include post-training enhancements, the decrease will be smaller (as [reason], perhaps ~0.5. This reduces my median estimate to r = ~1.2 (1.7-0.5).

  Overall, my median estimate of r is 1.2. I use a log-uniform distribution with the bounds 3X higher and lower (0.4 to 3.6).

• Tom DavidsonJan 22, 2025, 4:33 PM

  LW: 7 AF: 3

  0

  AF

  in reply to: ryan_greenblatt’s comment on: ryan_green­blatt’s Shortform

  I’ll paste my own estimate for this param in a different reply.

  But here are the places I most differ from you:

  • Bigger adjustment for ‘smarter AI’. You’ve argue in your appendix that, only including ‘more efficient’ and ‘faster’ AI, you think the software-only singularity goes through. I think including ‘smarter’ AI makes a big difference. This evidence suggests that doubling training FLOP doubles output-per-FLOP 1-2 times. In addition, algorithmic improvements will improve runtime efficiency. So overall I think a doubling of algorithms yields ~two doublings of (parallel) cognitive labour.

    • --> software singularity more likely

  • Lower lambda. I’d now use more like lambda = 0.4 as my median. There’s really not much evidence pinning this down; I think Tamay Besiroglu thinks there’s some evidence for values as low as 0.2. This will decrease the observed historical increase in human workers more than it decreases the gains from algorithmic progress (bc of speed improvements)

    • --> software singularity slightly more likely

  • Complications thinking about compute which might be a wash.

    • Number of useful-experiments has increased by less than 4X/​year. You say compute inputs have been increasing at 4X. But simultaneously the scale of experiments ppl must run to be near to the frontier has increased by a similar amount. So the number of near-frontier experiments has not increased at all.

      • This argument would be right if the ‘usefulness’ of an experiment depends solely on how much compute it uses compared to training a frontier model. I.e. experiment_usefulness = log(experiment_compute /​ frontier_model_training_compute). The 4X/​year increases the numerator and denominator of the expression, so there’s no change in usefulness-weighted experiments.

      • That might be false. GPT-2-sized experiments might in some ways be equally useful even as frontier model size increases. Maybe a better expression would be experiment_usefulness = alpha * log(experiment_compute /​ frontier_model_training_compute) + beta * log(experiment_compute). In this case, the number of usefulness-weighted experiments has increased due to the second term.

      • --> software singularity slightly more likely

    • Steeper diminishing returns during software singularity. Recent algorithmic progress has grabbed low-hanging fruit from new hardware scales. During a software-only singularity that won’t be possible. You’ll have to keep finding new improvements on the same hardware scale. Returns might diminish more quickly as a result.

      • --> software singularity slightly less likely

    • Compute share might increase as it becomes scarce. You estimate a share of 0.4 for compute, which seems reasonable. But it might fall over time as compute becomes a bottleneck. As an intuition pump, if your workers could think 1e10 times faster, you’d be fully constrained on the margin by the need for more compute: more labour wouldn’t help at all but more compute could be fully utilised so the compute share would be ~1.

      • --> software singularity slightly less likely

    • --> overall these compute adjustments prob make me more pessimistic about the software singularity, compared to your assumptions

  Taking it all together, i think you should put more probability on the software-only singluarity, mostly because of capability improvements being much more significant than you assume.

Hu­man takeover might be worse than AI takeover

• Tom DavidsonDec 20, 2024, 9:17 AM

  1 point

  0

  on: Should there be just one west­ern AGI pro­ject?

  One idea that seems potentially promising is to have a single centralised project and minimize the chance it becomes too powerful by minimizing its ability to take actions in the broader world.

  Concretely, a ‘Pre-Training Project’ does pre-training and GCR safety assessment, post-training needed for the above activities (including post-training to make AI R&D agents and evaluating the safety of post-training techniques), and nothing else. And then have many (>5) companies that do fine-tuning, scaffolding, productising, selling API access, and use-case-specific safety assessments.

  Why is this potentially the best of both worlds?

  • Much less concentration of power. The Pre-Training Project is strictly banned from these further activities (and indeed from any other activities) and it is closely monitored. This significantly reduces the (massive and very problematic) concentration of power you’d get from just one project selling AGI services to the world. It can’t shape the uses of the technology to its own private benefit, can’t charge monopoly prices, can’t use its superhuman AI and massive profits for political lobbying and shaping public opinion. Instead, multiple private companies will compete to ensure that the rest of the world gets maximum benefit from the tech.

    • More work is needed to see whether the power of the Pre-Training Project could really be robustly limited in this way.

  • No ‘race to the bottom’ within the west. Only one project is allowed to increase the effective compute used in pre-training. It’s not racing with other Western projects, so there is no ‘race to the bottom’. (Though obviously international racing here could still be a problem.)

• Tom DavidsonDec 11, 2024, 2:46 PM

  1 point

  0

  in reply to: Aaron_Scher’s comment on: Should there be just one west­ern AGI pro­ject?

  You could find a way of proving to the world that your AI is aligned, which other labs can’t replicate, giving you economic advantage.

  
  
  I don’t expect this to be a very large effect. It feels similar to an argument like “company A will be better on ESG dimensions and therefore more and customers will switch to using it”. Doing a quick review of the literature on that, it seems like there’s a small but notable change in consumer behavior for ESG-labeled products.

  It seems quite different to the ESG case. Customers don’t personally benefit from using a company with good ESG. They will benefit from using an aligned AI over a misaligned one.

  In the AI space, it doesn’t seem to me like any customers care about OpenAI’s safety team disappearing (except a few folks in the AI safety world).

  Again though, customers currently have no selfish reason to care.

  In this particular case, I expect the technical argument needed to demonstrate that some family of AI systems are aligned while others are not is a really complicated argument; I expect fewer than 500 people would be able to actually verify such an argument (or the initial “scalable alignment solution”), maybe zero people.

  It’s quite common for only a very small number of ppl to have the individual ability to verify a safety case, but many more to defer to their judgement. People may defer to an AISI, or a regulatory agency.

• Tom DavidsonDec 9, 2024, 5:19 PM

  3 points

  2

  in reply to: Rohin Shah’s comment on: Should there be just one west­ern AGI pro­ject?

  Fwiw, my own position is that for both infosec and racing it’s the brute fact that USG see fits to centralise all resources and develop AGI asap that would cause China to 1) try much harder to steal the weights than when private companies had developed the same capabilities themselves, 2) try much harder to race to AGI themselves.

• Tom DavidsonDec 9, 2024, 5:13 PM

  2 points

  0

  in reply to: Aaron_Scher’s comment on: Should there be just one west­ern AGI pro­ject?

  Quick clarification on terminology. We’ve used ‘centralised’ to mean “there’s just one project doing pre-training”. So having regulations that enforce good safety practice or gate-keep new training runs don’t count. I think this is a more helpful use of the term. It directly links to the power concentration concerns we’ve raised. I think the best versions of non-centralisation will involve regulations like these but that’s importantly different from one project having sole control of an insanely powerful technology.

  Compelling experimental evidence

  Currently there’s no basically no empirical evidence that misaligned power-seeking emerges by default, let alone scheming. If we got strong evidence that scheming happens by default then I expect that all projects would do way more work to check for and avoid scheming, whether centralised or not. Attitudes change on all levels: project technical staff, technical leadership, regulators, open-source projects.

  You can also iterate experimentally to understand the conditions that cause scheming, allowing empirical progress on scheming like was never before possible.

  This seems like a massive game changer to me. I truly believe that if we picked one of today’s top-5 labs at random and all the others were closed, this would be meaningfully less likely to happen and that would be a big shame.
  

  Scalable alignment solution

  You’re right there’s IP reasons against sharing. I believe it would be in line with many company’s missions to share, but they may not. Even so, there’s a lot you can do with aligned AGI. You could use it to produce compelling evidence about whether other AIs are aligned. You could find a way of proving to the world that your AI is aligned, which other labs can’t replicate, giving you economic advantage. It would be interesting to explore threats models where AI takes over despite a project solving this, and it doesn’t seem crazy, but i’d predict that we’d conclude the odds are better than if there’s 5 projects of which 2 have solved it than if there’s one project with a ²⁄₅ chance of success.

  RSPs

  Maybe you think everything is hopeless unless there are fundamental breakthroughs? My view is that we face severe challenges ahead, and have very tough decisions to make. But I believe that a highly competent and responsible project could likely find a way to leverage AI systems to solve AI alignment safely. Doing this isn’t just about “having the right values”. It’s much more about being highly competent, focussed on what really matters, prioritising well, and having good processes. If just one lab figures out how to do this all in a way that is commercially competitive and viable, that’s a proof of concept that developing AGI safety is possible. Excuses won’t work for other labs, as we can say “well lab X did it”.

  Overall

  I’m not confident “one apple saves the bunch”. But I expect most ppl on LW to assume “one apple spoils the bunch” and i think the alternative perspective is very underrated. My synthesis would probably be that at at current capability levels and in the next few years “one apple saves the bunch” wins by a large margin, but that at some point when AI is superhuman it could easily reverse bc AI gets powerful enough to design world-ending WMDs.

  (Also, i wanted to include this debate in the post but we felt it would over-complicate things. I’m glad you raised it and strongly upvoted your initial comment.)

• Tom DavidsonDec 6, 2024, 2:34 PM

  3 points

  0

  in reply to: Aaron_Scher’s comment on: Should there be just one west­ern AGI pro­ject?

  I agree with Rose’s reply, and would go further. I think there are many actions that just one responsible lab could take that would completely change the game board:

  • Find and share a scalable solution to alignment

  • Provide compelling experimental evidence that standard training methods lead to misaligned power-seeking AI by default

  • Develop and share best practices for responsible scaling that are both commercially viable and safe.

  You comment argues that “one bad apple spoils the bunch”, but it’s also plausible that “one good apple saves the bunch”

• Tom DavidsonDec 6, 2024, 2:27 PM

  1 point

  0

  in reply to: Gurkenglas’s comment on: Should there be just one west­ern AGI pro­ject?

  I think the argument for combining separate US and Chinese projects into one global project is probably stronger than the argument for centralising US development. That’s because racing between US companies can potentially be handled by USG regulation, but racing between US and China can’t be similarly handled.

  OTOH, the ‘info security’ benefits of centralisation mostly wouldn’t apply