Teortaxes: it’s about time ML twitter got brought up to speed on what “takeoff speeds” mean. Christiano: “There will be a complete 4 year interval in which world output doubles, before the first 1 year interval in which world output doubles.” That’s slow. We’re in the early stages of it.
I continue to think that by this definition takeoff will be fast, not slow. I think recent progress is making my prediction here look more and more likely, because it is making timelines seem shorter and shorter. (Once we get superintelligence, world output will be doubling in a year or less, I think. So slow takeoff by Paul’s definition is what happens when powerful pre-ASI systems are nevertheless widely deployed in the economy for long enough to double it. If, like me, you think that ASI is just a few years away, then there isn’t much time left for pre-ASI systems to explode into the economy and double it.)
I think on a strict interpretation of Christiano’s definition, we’re almost right on the bubble. Suppose we were to take something like Nvidia’s market cap as a very loose proxy for overall growth in accumulated global wealth due to AI. If it keeps doubling or tripling annually, but the return on that capital stays around 4-5% (aka if we assume the market prices things mostly correctly), then there would be a 4 year doubling just before the first 1 year doubling. But if it quadruples or faster annually, there won’t be. Note: my math is probably wrong here, and the metric I’m suggesting is definitely wrong, but I don’t think that affects my thinking on this in principle. I’m sure with some more work I could figure out the exact rate at which the exponent of an exponential can linearly increase while still technically complying with Christiano’s definition.
But really, I don’t think this kind of fine splitting rolls with the underlying differences that distinguish fast and slow takeoff as their definers originally intended. I suspect if we do have a classically-defined “fast” takeoff it’ll never be reflected in GDP or asset market price statistics at all, because the metric will be obsolete before the data is collected.
Note: I don’t actually have a strong opinion or clear preference on whether takeoff will remain smooth or become sharp in this sense.
This is what matters for AI R&D speed and for almost all recursive self-improvement.
Zvi is not quite correct when he is saying
If o3 was as good on most tasks as it is at coding or math, then it would be AGI.
o3 is not that good in coding and math (e.g. it only gets 71.7% on SWE-bench verified), it is not a “narrow AGI” yet. But it is strong enough, it’s a giant step forward.
For example, if one takes Sakana’s “AI scientist”, upgrades it slightly, and uses o3 as a back-end, it is likely that one can generate NeurIPS/ICLR quality papers and as many of those as one wants.
So, another upgrade (or a couple of upgrades) beyond o3, and we will reach that coveted “narrow AGI” stage.
What OpenAI has demonstrated is that it is much easier to achieve “narrow AGI” than “full AGI”. This does suggest a road to ASI without going through anything remotely close to a “full AGI” stage, with missing capabilities to be filled afterwards.
I continue to think that by this definition takeoff will be fast, not slow. I think recent progress is making my prediction here look more and more likely, because it is making timelines seem shorter and shorter. (Once we get superintelligence, world output will be doubling in a year or less, I think. So slow takeoff by Paul’s definition is what happens when powerful pre-ASI systems are nevertheless widely deployed in the economy for long enough to double it. If, like me, you think that ASI is just a few years away, then there isn’t much time left for pre-ASI systems to explode into the economy and double it.)
I think on a strict interpretation of Christiano’s definition, we’re almost right on the bubble. Suppose we were to take something like Nvidia’s market cap as a very loose proxy for overall growth in accumulated global wealth due to AI. If it keeps doubling or tripling annually, but the return on that capital stays around 4-5% (aka if we assume the market prices things mostly correctly), then there would be a 4 year doubling just before the first 1 year doubling. But if it quadruples or faster annually, there won’t be. Note: my math is probably wrong here, and the metric I’m suggesting is definitely wrong, but I don’t think that affects my thinking on this in principle. I’m sure with some more work I could figure out the exact rate at which the exponent of an exponential can linearly increase while still technically complying with Christiano’s definition.
But really, I don’t think this kind of fine splitting rolls with the underlying differences that distinguish fast and slow takeoff as their definers originally intended. I suspect if we do have a classically-defined “fast” takeoff it’ll never be reflected in GDP or asset market price statistics at all, because the metric will be obsolete before the data is collected.
Note: I don’t actually have a strong opinion or clear preference on whether takeoff will remain smooth or become sharp in this sense.
Right. We should probably introduce a new name, something like narrow AGI, to denote a system which is AGI-level in coding and math.
This kind of system will be “AGI” as redefined by Tom Davidson in https://www.lesswrong.com/posts/Nsmabb9fhpLuLdtLE/takeoff-speeds-presentation-at-anthropic:
This is what matters for AI R&D speed and for almost all recursive self-improvement.
Zvi is not quite correct when he is saying
o3 is not that good in coding and math (e.g. it only gets 71.7% on SWE-bench verified), it is not a “narrow AGI” yet. But it is strong enough, it’s a giant step forward.
For example, if one takes Sakana’s “AI scientist”, upgrades it slightly, and uses o3 as a back-end, it is likely that one can generate NeurIPS/ICLR quality papers and as many of those as one wants.
So, another upgrade (or a couple of upgrades) beyond o3, and we will reach that coveted “narrow AGI” stage.
What OpenAI has demonstrated is that it is much easier to achieve “narrow AGI” than “full AGI”. This does suggest a road to ASI without going through anything remotely close to a “full AGI” stage, with missing capabilities to be filled afterwards.