Ajeya’s timelines report is the best thing that’s ever been written about AI timelines imo. Whenever people ask me for my views on timelines, I go through the following mini-flowchart:
1. Have you read Ajeya’s report?
--If yes, launch into a conversation about the distribution over 2020′s training compute and explain why I think the distribution should be substantially to the left, why I worry it might shift leftward faster than she projects, and why I think we should use it to forecast AI-PONR instead of TAI.
--If no, launch into a conversation about Ajeya’s framework and why it’s the best and why all discussion of AI timelines should begin there.
So, why do I think it’s the best? Well, there’s a lot to say on the subject, but, in a nutshell: Ajeya’s framework is to AI forecasting what actual climate models are to climate change forecasting (by contrast with lower-tier methods such as “Just look at the time series of temperature over time / AI performance over time and extrapolate” and “Make a list of factors that might push the temperature up or down in the future / make AI progress harder or easier,” and of course the classic “poll a bunch of people with vaguely related credentials.”
There’s something else which is harder to convey… I want to say Ajeya’s model doesn’t actually assume anything, or maybe it makes only a few very plausible assumptions. This is underappreciated, I think. People will say e.g. “I think data is the bottleneck, not compute.” But Ajeya’s model doesn’t assume otherwise! If you think data is the bottleneck, then the model is more difficult for you to use and will give more boring outputs, but you can still use it. (Concretely, you’d have 2020′s training compute requirements distribution with lots of probability mass way to the right, and then rather than say the distribution shifts to the left at a rate of about one OOM a decade, you’d input whatever trend you think characterizes the likely improvements in data gathering.)
The upshot of this is that I think a lot of people are making a mistake when they treat Ajeya’s framework as just another model to foxily aggregate over. “When I think through Ajeya’s model, I get X timelines, but then when I extrapolate out GWP trends I get Y timelines, so I’m going to go with (X+Y)/2.” I think instead everyone’s timelines should be derived from variations on Ajeya’s model, with extensions to account for things deemed important (like data collection progress) and tweaks upwards or downwards to account for the rest of the stuff not modelled.
Ajeya’s timelines report is the best thing that’s ever been written about AI timelines imo. Whenever people ask me for my views on timelines, I go through the following mini-flowchart:
1. Have you read Ajeya’s report?
--If yes, launch into a conversation about the distribution over 2020′s training compute and explain why I think the distribution should be substantially to the left, why I worry it might shift leftward faster than she projects, and why I think we should use it to forecast AI-PONR instead of TAI.
--If no, launch into a conversation about Ajeya’s framework and why it’s the best and why all discussion of AI timelines should begin there.
So, why do I think it’s the best? Well, there’s a lot to say on the subject, but, in a nutshell: Ajeya’s framework is to AI forecasting what actual climate models are to climate change forecasting (by contrast with lower-tier methods such as “Just look at the time series of temperature over time / AI performance over time and extrapolate” and “Make a list of factors that might push the temperature up or down in the future / make AI progress harder or easier,” and of course the classic “poll a bunch of people with vaguely related credentials.”
There’s something else which is harder to convey… I want to say Ajeya’s model doesn’t actually assume anything, or maybe it makes only a few very plausible assumptions. This is underappreciated, I think. People will say e.g. “I think data is the bottleneck, not compute.” But Ajeya’s model doesn’t assume otherwise! If you think data is the bottleneck, then the model is more difficult for you to use and will give more boring outputs, but you can still use it. (Concretely, you’d have 2020′s training compute requirements distribution with lots of probability mass way to the right, and then rather than say the distribution shifts to the left at a rate of about one OOM a decade, you’d input whatever trend you think characterizes the likely improvements in data gathering.)
The upshot of this is that I think a lot of people are making a mistake when they treat Ajeya’s framework as just another model to foxily aggregate over. “When I think through Ajeya’s model, I get X timelines, but then when I extrapolate out GWP trends I get Y timelines, so I’m going to go with (X+Y)/2.” I think instead everyone’s timelines should be derived from variations on Ajeya’s model, with extensions to account for things deemed important (like data collection progress) and tweaks upwards or downwards to account for the rest of the stuff not modelled.