Thanks for doing this! While I have massive disagreements with the content, this was exactly the sort of thing I hoped to inspire with What 2026 Looks Like. Well done! I hope that others follow your lead.
Thoughts/comments:
1. What you’ve written happening in this decade is basically what I’d expect to happen if progress towards AGI came to a complete halt today, and all that people were allowed to do was small fine-tuning runs of existing models. Instead I think we’ll get all the way to APS-AI. You have what appear to be APS-AIs first showing up in the 2030s. Why not the 20s? Is it because they need to have more parameters, training time, etc. than is feasible in the 30s? Because e.g. they need to be 100T+ parameters trained compute-optimally where each data point is a many-step trajectory? 2. You speak of research being increasingly automated, e.g. coding productivity doubling, language-model-powered search being much better… Why hasn’t this accelerated progress towards AGI / APS-AI? I’d argue that progress will start noticeably accelerating in the twenties. 3. Why is there no singularity in this story? It looks like your story has APS-AI appearing in the thirties, and also “Chatbots have long passed all kinds of Turing tests and start to fuck around on the internet.” And of course they have automated most AI research and large parts of the economy. Why does it take 10-20 more years to get to superintelligence?
I think it’s mostly my skepticism about extremely fast economic transformations.
Like GPT-3 could probably automate more parts of the economy today but somehow it just takes a while for people to understand that and get it to work in practice. I also expect that it will take a couple of years between showing the capabilities of new AI systems in the lab and widespread economic impact just because humans take a while to adapt (at least with narrow systems).
At some point (maybe in 2030) we will reach a level where AI is as capable as humans in many tasks and then the question is obviously how fast it can self-improve. I’m skeptical that it is possible to self-improve as fast as the classic singularity story would suggest. In my mind, you mostly need more compute for training, new training data, new task design, etc. I think it will take some time for the AI to come up with all of that and even then, exponential demands just have their limits. Maybe the AI can get 100x compute and train a new model but getting 10000x compute probably won’t happen immediately (at least in my mind; arguments for or against are welcome).
Lastly, I wrote a story about my median scenario. I do have lots of uncertainty about how the TAI distribution should look like (see here) but my mode is at 2032-2035 (i.e. earlier than my median). So I could have also written a story with faster developments and it would reflect a slightly different corner of my probability distribution. But due to the reasons above, it would mostly look like a slightly faster version of this story.
And your scenario is within the space of scenarios that I think could happen, I just think it’s less likely than a less accelerationist, slower transition. But obviously not very confident in this prediction.
For pre-AGI systems, I agree that it’s going to take a substantial amount of time for them to be deployed as products and transform the economy. Which is why my What 2026 Looks Like story doesn’t have GDP even begin to accelerate by 2026. For post-AGI systems, I forecast a fairly quick FOOM period (less than a year probably? Weeks or days, possibly) during which we keep basically the same hardware, but the “software” improves dramatically. We end up with something like von Neumann’s brain, but qualitatively better in every way, and also cheaper and faster, and with a huge population size (millions? Billions?). They do more quality-adjusted intellectual labor in a few hours than all of human science could do in decades. They figure out a plan for dramatically better hardware, dramatically better robots, etc. and then from then on the human economy works feverishly to follow The Plan as fast as possible, instructed by superintelligent AI overseers. I think maybe it’ll take only a few years at most to get to crazy nanobot tech (or some other kind of fully automated industry). Could take only a few days potentially.
I feel confused about your “pre-AGI”/”post-AGI” distinction. I expect that there will be a period of months or even years during which whether or not we’ve built “AGI” is up for debate. Given this, it feels very odd to say that takeoff might happen weeks after reaching AGI, because the takeoff period would then be much shorter than the uncertainty period.
By AGI I mean AI systems which can do every relevant intellectual task human professionals can do, only cheaper and faster. Because of variation, by the time we get AGI, we’ll have AI systems which are strongly superhuman at many relevant intellectual tasks. I feel fairly confident that by the time AGI exists, we’ll be months away from superintelligence at most, and possibly just hours. Absent defeaters such as the relevant powers coordinating to slow down the R&D. Main alternative is if the only available ways to significantly improve intelligence in the bottleneck dimensions is to do larger, longer training runs. Even then, months seems like a plausible timeframe, though admittedly it could take maybe two or three years. I’m not sure I agree with your expectation. I do think there’ll be lots of FUD and uncertainty around AGI, there already is. But this is consistent with the above claims.
I think this just isn’t a very helpful definition of AGI, and one which will likely lead people to misinterpret your statements, because it’s so sensitive to the final tasks automated (which might be totally uninteresting). Under this definition time to AGI, and time from AGI to superintelligence, might vary dramatically depending on what you count as an intellectual task.
Hmmm. The phrase is “relevant intellectual tasks.” You are saying people will prematurely declare that AGI has been achieved, months or even years before I would declare it, because they’ll classify as nonrelevant some task which I classify as relevant? (And which AIs still cannot do?) I am skeptical that this will be a problem in practice.
ETA: Also, I’d be interested to hear which alternative definitions you like better! I’m not particularly wedded to this one, I just think it’s better than various other definitions of AGI and waaaay better than TAI or GDP-based definitions.
Centrally, I’m thinking about big important things, like taking over the world, or making R&D go FOOM resulting in some other AI system which can take over the world. But I’m happy to have a broader conception of relevance on a case-by-case basis. Insofar as people have a broader conception of relevance than me, then that means AGI-by-their-definition might come hours or even several months later than AGI-by-my-definition. (The latter would happen in cases where R&D ability is significantly harder than world-takeover-ability. I guess in principle I could see this even resulting in a several-year gap, though I think that’s pretty unlikely.)
Can you provide some of your reasons or intuitions for this fast FOOM?
My intuition against it is mostly like “intelligence just seems to be compute bound and thus extremely fast takeoffs (hours to weeks) are unlikely”. But I feel very uncertain about this take and would like to refine it. So just understanding your intuitions better would probably already help a lot.
Every time I sit down to make a model of takeoff, or read someone else’s model & input my own values for the parameters, it ends up being pretty fast. Much faster than your story. (In fact, I’m not sure I’ve ever seen a model that results in a takeoff as slow as your story, even with other people’s values to the parameters.) That’s the main reason why I have faster-takeoff views.
There’s a big gap between “hours to weeks” and “10+ years!” I don’t think intelligence is compute bounded in the relevant sense, but even if it was (see my “Main alternative” in response to Richard elsewhere in this thread) it would maybe get us to a 3 year post-AGI takeoff at most, I’d say.
If you have time to elaborate more on your model—e.g. what you mean by intelligence being compute bounded, and how that translates into numbers for post-AGI takeoff speed—I’d be interested to hear it!
Basically, he has to convince rational skeptics of AGI changing our very way of life, and partially it’s to model what would happen even in the average case, I.e even if certain things go wrong, it still doesn’t undermine the story.
And that’s good, since most short timelines generally rely on most or all things coming to pass.
Thanks for doing this! While I have massive disagreements with the content, this was exactly the sort of thing I hoped to inspire with What 2026 Looks Like. Well done! I hope that others follow your lead.
Thoughts/comments:
1. What you’ve written happening in this decade is basically what I’d expect to happen if progress towards AGI came to a complete halt today, and all that people were allowed to do was small fine-tuning runs of existing models. Instead I think we’ll get all the way to APS-AI. You have what appear to be APS-AIs first showing up in the 2030s. Why not the 20s? Is it because they need to have more parameters, training time, etc. than is feasible in the 30s? Because e.g. they need to be 100T+ parameters trained compute-optimally where each data point is a many-step trajectory?
2. You speak of research being increasingly automated, e.g. coding productivity doubling, language-model-powered search being much better… Why hasn’t this accelerated progress towards AGI / APS-AI? I’d argue that progress will start noticeably accelerating in the twenties.
3. Why is there no singularity in this story? It looks like your story has APS-AI appearing in the thirties, and also “Chatbots have long passed all kinds of Turing tests and start to fuck around on the internet.” And of course they have automated most AI research and large parts of the economy. Why does it take 10-20 more years to get to superintelligence?
I think it’s mostly my skepticism about extremely fast economic transformations.
Like GPT-3 could probably automate more parts of the economy today but somehow it just takes a while for people to understand that and get it to work in practice. I also expect that it will take a couple of years between showing the capabilities of new AI systems in the lab and widespread economic impact just because humans take a while to adapt (at least with narrow systems).
At some point (maybe in 2030) we will reach a level where AI is as capable as humans in many tasks and then the question is obviously how fast it can self-improve. I’m skeptical that it is possible to self-improve as fast as the classic singularity story would suggest. In my mind, you mostly need more compute for training, new training data, new task design, etc. I think it will take some time for the AI to come up with all of that and even then, exponential demands just have their limits. Maybe the AI can get 100x compute and train a new model but getting 10000x compute probably won’t happen immediately (at least in my mind; arguments for or against are welcome).
Lastly, I wrote a story about my median scenario. I do have lots of uncertainty about how the TAI distribution should look like (see here) but my mode is at 2032-2035 (i.e. earlier than my median). So I could have also written a story with faster developments and it would reflect a slightly different corner of my probability distribution. But due to the reasons above, it would mostly look like a slightly faster version of this story.
And your scenario is within the space of scenarios that I think could happen, I just think it’s less likely than a less accelerationist, slower transition. But obviously not very confident in this prediction.
For pre-AGI systems, I agree that it’s going to take a substantial amount of time for them to be deployed as products and transform the economy. Which is why my What 2026 Looks Like story doesn’t have GDP even begin to accelerate by 2026. For post-AGI systems, I forecast a fairly quick FOOM period (less than a year probably? Weeks or days, possibly) during which we keep basically the same hardware, but the “software” improves dramatically. We end up with something like von Neumann’s brain, but qualitatively better in every way, and also cheaper and faster, and with a huge population size (millions? Billions?). They do more quality-adjusted intellectual labor in a few hours than all of human science could do in decades. They figure out a plan for dramatically better hardware, dramatically better robots, etc. and then from then on the human economy works feverishly to follow The Plan as fast as possible, instructed by superintelligent AI overseers. I think maybe it’ll take only a few years at most to get to crazy nanobot tech (or some other kind of fully automated industry). Could take only a few days potentially.
I feel confused about your “pre-AGI”/”post-AGI” distinction. I expect that there will be a period of months or even years during which whether or not we’ve built “AGI” is up for debate. Given this, it feels very odd to say that takeoff might happen weeks after reaching AGI, because the takeoff period would then be much shorter than the uncertainty period.
By AGI I mean AI systems which can do every relevant intellectual task human professionals can do, only cheaper and faster. Because of variation, by the time we get AGI, we’ll have AI systems which are strongly superhuman at many relevant intellectual tasks.
I feel fairly confident that by the time AGI exists, we’ll be months away from superintelligence at most, and possibly just hours. Absent defeaters such as the relevant powers coordinating to slow down the R&D. Main alternative is if the only available ways to significantly improve intelligence in the bottleneck dimensions is to do larger, longer training runs. Even then, months seems like a plausible timeframe, though admittedly it could take maybe two or three years.
I’m not sure I agree with your expectation. I do think there’ll be lots of FUD and uncertainty around AGI, there already is. But this is consistent with the above claims.
I think this just isn’t a very helpful definition of AGI, and one which will likely lead people to misinterpret your statements, because it’s so sensitive to the final tasks automated (which might be totally uninteresting). Under this definition time to AGI, and time from AGI to superintelligence, might vary dramatically depending on what you count as an intellectual task.
Hmmm. The phrase is “relevant intellectual tasks.” You are saying people will prematurely declare that AGI has been achieved, months or even years before I would declare it, because they’ll classify as nonrelevant some task which I classify as relevant? (And which AIs still cannot do?) I am skeptical that this will be a problem in practice.
ETA: Also, I’d be interested to hear which alternative definitions you like better! I’m not particularly wedded to this one, I just think it’s better than various other definitions of AGI and waaaay better than TAI or GDP-based definitions.
Relevant to what?
Centrally, I’m thinking about big important things, like taking over the world, or making R&D go FOOM resulting in some other AI system which can take over the world. But I’m happy to have a broader conception of relevance on a case-by-case basis. Insofar as people have a broader conception of relevance than me, then that means AGI-by-their-definition might come hours or even several months later than AGI-by-my-definition. (The latter would happen in cases where R&D ability is significantly harder than world-takeover-ability. I guess in principle I could see this even resulting in a several-year gap, though I think that’s pretty unlikely.)
Hmmm interesting.
Can you provide some of your reasons or intuitions for this fast FOOM?
My intuition against it is mostly like “intelligence just seems to be compute bound and thus extremely fast takeoffs (hours to weeks) are unlikely”. But I feel very uncertain about this take and would like to refine it. So just understanding your intuitions better would probably already help a lot.
Every time I sit down to make a model of takeoff, or read someone else’s model & input my own values for the parameters, it ends up being pretty fast. Much faster than your story. (In fact, I’m not sure I’ve ever seen a model that results in a takeoff as slow as your story, even with other people’s values to the parameters.) That’s the main reason why I have faster-takeoff views.
There’s a big gap between “hours to weeks” and “10+ years!” I don’t think intelligence is compute bounded in the relevant sense, but even if it was (see my “Main alternative” in response to Richard elsewhere in this thread) it would maybe get us to a 3 year post-AGI takeoff at most, I’d say.
If you have time to elaborate more on your model—e.g. what you mean by intelligence being compute bounded, and how that translates into numbers for post-AGI takeoff speed—I’d be interested to hear it!
Basically, he has to convince rational skeptics of AGI changing our very way of life, and partially it’s to model what would happen even in the average case, I.e even if certain things go wrong, it still doesn’t undermine the story.
And that’s good, since most short timelines generally rely on most or all things coming to pass.