My mainline prediction scenario for the next decades.
My mainline prediction * :
LLMs will not scale to AGI. They will not spawn evil gremlins or mesa-optimizers. BUT Scaling laws will continue to hold and future LLMs will be very impressive and make a sizable impact on the real economy and science over the next decade. EDIT: since there is a lot of confusion about this point. BY LLM I mean the paradigm of pre-trained transformers. This does not include different paradigms that follow pre-trained transformers but are still called large language models.
there is a single innovation left to make AGI-in-the-alex sense work, i.e. coherent, long-term planning agents (LTPA) that are effective and efficient in data sparse domains over long horizons.
that innovation will be found within the next 10-15 years
It will be clear to the general public that these are dangerous
governments will act quickly and (relativiely) decisively to bring these agents under state-control. national security concerns will dominate.
power will reside mostly with governments AI safety institutes and national security agencies. In so far as divisions of tech companies are able to create LTPAs they will be effectively nationalized.
International treaties will be made to constrain AI, outlawing the development of LTPAs by private companies. Great power competition will mean US and China will continue developing LTPAs, possibly largely boxed. Treaties will try to constrain this development with only partial succes (similar to nuclear treaties).
LLMs will continue to exist and be used by the general public
Conditional on AI ruin the closest analogy is probably something like the Cortez-Pizarro-Afonso takeovers. Unaligned AI will rely on human infrastructure and human allies for the earlier parts of takeover—but its inherent advantages in tech, coherence, decision-making and (artificial) plagues will be the deciding factor.
The world may be mildly multi-polar.
This will involve conflict between AIs.
AIs very possible may be able to cooperate in ways humans can’t.
The arrival of AGI will immediately inaugurate a scientific revolution. Sci-fi sounding progress like advanced robotics, quantum magic, nanotech, life extension, laser weapons, large space engineering, cure of many/most remaining diseases will become possible within two decades of AGI, possibly much faster.
Military power will shift to automated manufacturing of drones & weaponized artificial plagues. Drones, mostly flying will dominate the battlefield. Mass production of drones and their rapid and effective deployment in swarms will be key to victory.
Two points on which I differ with most commentators: (i) I believe AGI is a real (mostly discrete) thing , not a vibe, or a general increase of improved tools. I believe it is inherently agenctic. I don’t think spontaneous emergence of agents is impossible but I think it is more plausible agents will be built rather than grown.
(ii) I believe in general the ea/ai safety community is way overrating the importance of individual tech companies vis a vis broader trends and the power of governments. I strongly agree with Stefan Schubert’s take here on the latent hidden power of government: https://stefanschubert.substack.com/p/crises-reveal-centralisation
Consequently, the ea/ai safety community is often myopically focusing on boardroom politics that are relativily inconsequential in the grand scheme of things.
*where by mainline prediction I mean the scenario that is the mode of what I expect. This is the single likeliest scenario. However, since it contains a large number of details each of which could go differently, the probability on this specific scenario is still low.
governments will act quickly and (relativiely) decisively to bring these agents under state-control. national security concerns will dominate.
I dunno, like 20 years ago if someone had said “By the time somebody creates AI that displays common-sense reasoning, passes practically any written test up including graduate-level, (etc.), obviously governments will be flipping out and nationalizing AI companies etc.”, to me that would have seemed like a reasonable claim. But here we are, and the idea of the USA govt nationalizing OpenAI seems a million miles outside the Overton window.
Likewise, if someone said “After it becomes clear to everyone that lab leaks can cause pandemics costing trillions of dollars and millions of lives, then obviously governments will be flipping out and banning the study of dangerous viruses—or at least, passing stringent regulations with intrusive monitoring and felony penalties for noncompliance etc,” then that would also have sounded reasonable to me! But again, here we are.
So anyway, my conclusion is that when I ask my intuition / imagination whether governments will flip out in thus-and-such circumstance, my intuition / imagination is really bad at answering that question. I think it tends to underweight the force compelling goverments to continue following longstanding customs / habits / norms? Or maybe it’s just hard to predict and these are two cherrypicked examples, and if I thought a bit harder I’d come up with lots of examples in the opposite direction too (i.e., governments flipping out and violating longstanding customs on a dime)? I dunno. Does anyone have a good model here?
One strong reason to think the AI case might be different is that US national security will be actively using AI to build weapons and thus it will be relatively clear and salient to US national security when things get scary.
For another thing, I feel like there’s a normal playbook for new weapons-development technology, which is that the military says “Ooh sign me up”, and (in the case of the USA) the military will start using the tech in-house (e.g. at NRL) and they’ll also send out military contracts to develop the tech and apply it to the military. Those contracts are often won by traditional contractors like Raytheon, but in some cases tech companies might bid as well.
I can’t think of precedents where a tech was in wide use by the private sector but then brought under tight military control in the USA. Can you?
The closest things I can think of is secrecy orders (the US military gets to look at every newly-approved US patent and they can choose to declare them to be military secrets) and ITAR (the US military can declare that some area of tech development, e.g. certain types of high-quality IR detectors that are useful for night vision and targeting, can’t be freely exported, nor can their schematics etc. be shared with non-US citizens).
Like, I presume there are lots of non-US-citizens who work for OpenAI. If the US military were to turn OpenAI’s ongoing projects into classified programs (for example), those non-US employees wouldn’t qualify for security clearances. So that would basically destroy OpenAI rather than control it (and of course the non-USA staff would bring their expertise elsewhere). Similarly, if the military was regularly putting secrecy orders on OpenAI’s patents, then OpenAI would obviously respond by applying for fewer patents, and instead keeping things as trade secrets which have no normal avenue for military review.
By the way, fun fact: if some technology or knowledge X is classified, but X is also known outside a classified setting, the military deals with that in a very strange way: people with classified access to X aren’t allowed to talk about X publicly, even while everyone else in the world does! This comes up every time there’s a leak, for example (e.g. Snowden). I mention this fact to suggest an intuitive picture where US military secrecy stuff involves a bunch of very strict procedures that everyone very strictly follows even when they kinda make no sense.
(I have some past experience with ITAR, classified programs, and patent secrecy orders, but I’m not an expert with wide-ranging historical knowledge or anything like that.)
But here we are, and the idea of the USA govt nationalizing OpenAI seems a million miles outside the Overton window.
Registering that it does not seem that far out the Overton window to me anymore. My own advance prediction of how much governments would be flipping out around this capability level has certainly been proven a big underestimate.
I think this will look a bit outdated in 6-12 months, when there is no longer a clear distinction between LLMs and short term planning agents, and the distinction between the latter and LTPAs looks like a scale difference comparable to GPT2 vs GPT3 rather than a difference in kind. At what point do you imagine a national government saying “here but no further?”.
So you are predicting that within 6-12 months, there will no longer be a clear distinction between LLMs and “short term planning agents”. Do you mean that agentic LLM scaffolding like Auto-GPT will qualify as such?
I think scaffolding is the wrong metaphor. Sequences of actions, observations and rewards are just more tokens to be modeled, and if I were running Google I would be busy instructing all work units to start packaging up such sequences of tokens to feed into the training runs for Gemini models. Many seemingly minor tasks (e.g. app recommendation in the Play store) either have, or could have, components of RL built into the pipeline, and could benefit from incorporating LLMs, either by putting the RL task in-context or through fine-tuning of very fast cheap models.
So when I say I don’t see a distinction between LLMs and “short term planning agents” I mean that we already know how to subsume RL tasks into next token prediction, and so there is in some technical sense already no distinction. It’s a question of how the underlying capabilities are packaged and deployed, and I think that within 6-12 months there will be many internal deployments of LLMs doing short sequences of tasks within Google. If that works, then it seems very natural to just scale up sequence length as generalisation improves.
Arguably fine-tuning a next-token predictor on action, observation, reward sequences, or doing it in-context, is inferior to using algorithms like PPO. However, the advantage of knowledge transfer from the rest of the next-token predictor’s data distribution may more than compensate for this on some short-term tasks.
I think o1 is a partial realization of your thesis, and the only reason it’s not more successful is because the compute used for GPT-o1 and GPT-4o were essentially the same:
As far as I can tell Strawberry is proving me right: it’s going beyond pre-training and scales inference—the obvious next step.
A lot of people said just scaling pre-trained transformers would scale to AGI. I think that’s silly and doesn’t make sense. But now you don’t have to believe me—you can just use OpenAIs latest model.
The next step is to do efficient long-horizon RL for data-sparse domains.
Strawberry working suggest that this might not be so hard. Don’t be fooled by the modest gains of Strawberry so far. This is a new paradigm that is heading us toward true AGI and superintelligence.
Yeah actually Alexander and I talked about that briefly this morning. I agree that the crux is “does this basic kind of thing work” and given that the answer appears to be “yes” we can confidently expect scale (in both pre-training and inference compute) to deliver significant gains.
I’d love to understand better how the RL training for CoT changes the representations learned during pre-training.
in my reading, Strawberry is showing that indeed scaling just pretraining transformers will *not* lead to AGI. The new paradigm is inference-scaling—the obvious next step is doing RL on long horizons and sparse data domains. I have been saying this ever since gpt-3 came out.
For the question of general intelligence imho the scaling is conceptually a red herring: any (general purpose) algorithm will do better when scaled. The key in my mind is the algorithm not the resource, just like I would say a child is generally intelligent while a pocket calculator is not even if the child can’t count to 20 yet. It’s about the meta-capability to learn not the capability.
As we spoke earlier—it was predictable that this was going to be the next step. It was likely it was going to work, but there was a hopeful world in which doing the obvious thing turned out to be harder. That hope has been dashed—it suggests longer horizons might be easy too. This means superintelligence within two years isnot out of the question.
We have been shown that this search algorithm works, and we not yet have been shown that the other approaches don’t work.
Remember, technological development is disjunctive, and just because you’ve shown that 1 approach works, doesn’t mean that we have been shown that only that approach works.
Of course, people will absolutely try to scale this one up now that they found success, and I think that timelines have definitely been shortened, but remember that AI progress is closer to a disjunctive scenario than conjunctive scenario:
I agree with this quote below, but I wanted to point out the disjunctiveness of AI progress:
As we spoke earlier—it was predictable that this was going to be the next step. It was likely it was going to work, but there was a hopeful world in which doing the obvious thing turned out to be harder. That hope has been dashed—it suggests longer horizons might be easy too. This means superintelligence within two years is not out of the question.
strong disagree. i would be highly surprised if there were multiple essentially different algorithms to achieve general intelligence*.
I also agree with the Daniel Murfet’s quote. There is a difference between a disjunction before you see the data and a disjunction after you see the data. I agree AI development is disjunctive before you see the data—but in hindsight all the things that work are really minor variants on a single thing that works.
*of course “essentially different” is doing a lot of work here. some of the conceptual foundations of intelligence haven’t been worked out enough (or Vanessa has and I don’t understand it yet) for me to make a formal statement here.
Re different algorithms, I actually agree with both you and Daniel Murfet in that conditional on non-reversible computers, there is at most 1-3 algorithms to achieve intelligence that can scale arbitrarily large, and I’m closer to 1 than 3 here.
But once reversible computers/superconducting wires are allowed, all bets are off on how many algorithms are allowed, because you can have far, far more computation with far, far less waste heat leaving, and a lot of the design of computers is due to heat requirements.
Reversible computing and superconducting wires seem like hardware innovations. You are saying that this will actually materially change the nature of the algorithm you’d want to run?
I’d bet against. I’d be surprised if this was the case. As far as I can tell everything we have so seen so far points to a common simple core of general intelligence algorithm (basically an open-loop RL algorithm on top of a pre-trained transformers). I’d be surprised if there were materially different ways to do this. One of the main takeaways of the last decade of deep learning process is just how little architecture matters—it’s almost all data and compute (plus I claim one extra ingredient, open-loop RL that is efficient on long horizons and sparse data novel domains)
I don’t know for certain of course. If I look at theoretical CS though the universality of computation makes me skeptical of radically different algorithms.
I’m a bit confused by what you mean by “LLMs will not scale to AGI” in combination with “a single innovation is all that is needed for AGI”.
E.g., consider the following scenarios:
AGI (in the sense you mean) is achieved by figuring out a somewhat better RL scheme and massively scaling this up on GPT-6.
AGI is achieved by doing some sort of architectural hack on top of GPT-6 which makes it able to reason in neuralese for longer and then doing a bunch of training to teach the model to use this well.
AGI is achieved via doing some sort of iterative RL/synth data/self-improvement process for GPT-6 in which GPT-6 generates vast amounts of synthetic data for itself using various tools.
IMO, these sound very similar to “LLMs scale to AGI” for many practical purposes:
LLM scaling is required for AGI
LLM scaling drives the innovation required for AGI
From the public’s perspective, it maybe just looks like AI is driven by LLMs getting better over time and various tweaks might be continuously introduced.
Maybe it is really key in your view that the single innovation is really discontinuous and maybe the single innovation doesn’t really require LLM scaling.
I think a single innovation left to create LTPA is unlikely because it runs contrary to the history of technology and of machine learning. For example, in the 10 years before AlphaGo and before GPT-4, several different innovations were required—and that’s if you count “deep learning” as one item. ChatGPT actually understates the number here because different components of the transformer architecture like attention, residual streams, and transformer++ innovations were all developed separately.
Then I think you should specify that progress within this single innovation could be continuous over years and include 10+ ML papers in sequence each developing some sub-innovation.
Agreed on all points except a couple of the less consequential, where I don’t disagree.
Strongest agreement: we’re underestimating the importance of governments for alignment and use/misuse. We haven’t fully updated from the inattentive world hypothesis. Governments will notice the importance of AGI before it’s developed, and will seize control. They don’t need to nationalize the corporations, they just need to have a few people embedded at theh company and demand on threat of imprisonment that they’re kept involved with all consequential decisions on its use. I doubt they’d even need new laws, because the national security implications are enormous. But if they need new laws, they’ll create them as rapidly as necessary. Hopping borders will be difficult, and just put a different government in control.
Strongest disagreement: I think it’s likely that zero breakthroughs are needed to add long term planning capabilities to LLM-based systems, and so long term planning agents (I like the terminology) will be present very soon, and improve as LLMs continue to improve. I have specific reasons for thinking this. I could easily be wrong, but I’m pretty sure that the rational stance is “maybe”. This maybe advances the timelines dramatically.
Also strongly agree on AGI as a relatively discontinuous improvement; I worry that this is glossed over in modern “AI safety” discussions, causing people to mistake controlling LLMs for aligning the AGIs we’ll create on top of them. AGI alignment requires different conceptual work.
So somebody gets an agent which efficiently productively indefinitely works on any specified goal, then they just let the government find out and take it? No countermeasures?
What “coherent, long-term planning agents” means, and what is possible with these agents, is not clear to me. How would they overcome lack of access to knowledge, as was highlighted by F.A. Hayek in “The Use of Knowledge in Society”? What actions would they plan? How would their planning come to replace humans’ actions? (Achieving control over some sectors of battlefields would only be controlling destruction, of course, it would not be controlling creation.)
Some discussion is needed that recognizes and takes into account differences among governance structures. What seems the most relevant to me are these cases: (1) totalitarian governments, (2) somewhat-free governments, (3) transnational corporations, (4) decentralized initiatives. This is a new kind of competition, but the results will be like with major wars: Resilient-enough groups will survive the first wave or new groups will re-form later, and ultimately the competition will be won by the group that outproduces the others. In each successive era, the group that outproduces the others will be the group that leaves people the freest.
My mainline prediction scenario for the next decades.
My mainline prediction * :
LLMs will not scale to AGI. They will not spawn evil gremlins or mesa-optimizers. BUT Scaling laws will continue to hold and future LLMs will be very impressive and make a sizable impact on the real economy and science over the next decade. EDIT: since there is a lot of confusion about this point. BY LLM I mean the paradigm of pre-trained transformers. This does not include different paradigms that follow pre-trained transformers but are still called large language models.
there is a single innovation left to make AGI-in-the-alex sense work, i.e. coherent, long-term planning agents (LTPA) that are effective and efficient in data sparse domains over long horizons.
that innovation will be found within the next 10-15 years
It will be clear to the general public that these are dangerous
governments will act quickly and (relativiely) decisively to bring these agents under state-control. national security concerns will dominate.
power will reside mostly with governments AI safety institutes and national security agencies. In so far as divisions of tech companies are able to create LTPAs they will be effectively nationalized.
International treaties will be made to constrain AI, outlawing the development of LTPAs by private companies. Great power competition will mean US and China will continue developing LTPAs, possibly largely boxed. Treaties will try to constrain this development with only partial succes (similar to nuclear treaties).
LLMs will continue to exist and be used by the general public
Conditional on AI ruin the closest analogy is probably something like the Cortez-Pizarro-Afonso takeovers. Unaligned AI will rely on human infrastructure and human allies for the earlier parts of takeover—but its inherent advantages in tech, coherence, decision-making and (artificial) plagues will be the deciding factor.
The world may be mildly multi-polar.
This will involve conflict between AIs.
AIs very possible may be able to cooperate in ways humans can’t.
The arrival of AGI will immediately inaugurate a scientific revolution. Sci-fi sounding progress like advanced robotics, quantum magic, nanotech, life extension, laser weapons, large space engineering, cure of many/most remaining diseases will become possible within two decades of AGI, possibly much faster.
Military power will shift to automated manufacturing of drones & weaponized artificial plagues. Drones, mostly flying will dominate the battlefield. Mass production of drones and their rapid and effective deployment in swarms will be key to victory.
Two points on which I differ with most commentators: (i) I believe AGI is a real (mostly discrete) thing , not a vibe, or a general increase of improved tools. I believe it is inherently agenctic. I don’t think spontaneous emergence of agents is impossible but I think it is more plausible agents will be built rather than grown.
(ii) I believe in general the ea/ai safety community is way overrating the importance of individual tech companies vis a vis broader trends and the power of governments. I strongly agree with Stefan Schubert’s take here on the latent hidden power of government: https://stefanschubert.substack.com/p/crises-reveal-centralisation
Consequently, the ea/ai safety community is often myopically focusing on boardroom politics that are relativily inconsequential in the grand scheme of things.
*where by mainline prediction I mean the scenario that is the mode of what I expect. This is the single likeliest scenario. However, since it contains a large number of details each of which could go differently, the probability on this specific scenario is still low.
I dunno, like 20 years ago if someone had said “By the time somebody creates AI that displays common-sense reasoning, passes practically any written test up including graduate-level, (etc.), obviously governments will be flipping out and nationalizing AI companies etc.”, to me that would have seemed like a reasonable claim. But here we are, and the idea of the USA govt nationalizing OpenAI seems a million miles outside the Overton window.
Likewise, if someone said “After it becomes clear to everyone that lab leaks can cause pandemics costing trillions of dollars and millions of lives, then obviously governments will be flipping out and banning the study of dangerous viruses—or at least, passing stringent regulations with intrusive monitoring and felony penalties for noncompliance etc,” then that would also have sounded reasonable to me! But again, here we are.
So anyway, my conclusion is that when I ask my intuition / imagination whether governments will flip out in thus-and-such circumstance, my intuition / imagination is really bad at answering that question. I think it tends to underweight the force compelling goverments to continue following longstanding customs / habits / norms? Or maybe it’s just hard to predict and these are two cherrypicked examples, and if I thought a bit harder I’d come up with lots of examples in the opposite direction too (i.e., governments flipping out and violating longstanding customs on a dime)? I dunno. Does anyone have a good model here?
One strong reason to think the AI case might be different is that US national security will be actively using AI to build weapons and thus it will be relatively clear and salient to US national security when things get scary.
For one thing, COVID-19 obviously had impacts on military readiness and operations, but I think that fact had very marginal effects on pandemic prevention.
For another thing, I feel like there’s a normal playbook for new weapons-development technology, which is that the military says “Ooh sign me up”, and (in the case of the USA) the military will start using the tech in-house (e.g. at NRL) and they’ll also send out military contracts to develop the tech and apply it to the military. Those contracts are often won by traditional contractors like Raytheon, but in some cases tech companies might bid as well.
I can’t think of precedents where a tech was in wide use by the private sector but then brought under tight military control in the USA. Can you?
The closest things I can think of is secrecy orders (the US military gets to look at every newly-approved US patent and they can choose to declare them to be military secrets) and ITAR (the US military can declare that some area of tech development, e.g. certain types of high-quality IR detectors that are useful for night vision and targeting, can’t be freely exported, nor can their schematics etc. be shared with non-US citizens).
Like, I presume there are lots of non-US-citizens who work for OpenAI. If the US military were to turn OpenAI’s ongoing projects into classified programs (for example), those non-US employees wouldn’t qualify for security clearances. So that would basically destroy OpenAI rather than control it (and of course the non-USA staff would bring their expertise elsewhere). Similarly, if the military was regularly putting secrecy orders on OpenAI’s patents, then OpenAI would obviously respond by applying for fewer patents, and instead keeping things as trade secrets which have no normal avenue for military review.
By the way, fun fact: if some technology or knowledge X is classified, but X is also known outside a classified setting, the military deals with that in a very strange way: people with classified access to X aren’t allowed to talk about X publicly, even while everyone else in the world does! This comes up every time there’s a leak, for example (e.g. Snowden). I mention this fact to suggest an intuitive picture where US military secrecy stuff involves a bunch of very strict procedures that everyone very strictly follows even when they kinda make no sense.
(I have some past experience with ITAR, classified programs, and patent secrecy orders, but I’m not an expert with wide-ranging historical knowledge or anything like that.)
‘when things get scary’ when then?
Registering that it does not seem that far out the Overton window to me anymore. My own advance prediction of how much governments would be flipping out around this capability level has certainly been proven a big underestimate.
I think this will look a bit outdated in 6-12 months, when there is no longer a clear distinction between LLMs and short term planning agents, and the distinction between the latter and LTPAs looks like a scale difference comparable to GPT2 vs GPT3 rather than a difference in kind. At what point do you imagine a national government saying “here but no further?”.
So you are predicting that within 6-12 months, there will no longer be a clear distinction between LLMs and “short term planning agents”. Do you mean that agentic LLM scaffolding like Auto-GPT will qualify as such?
I think scaffolding is the wrong metaphor. Sequences of actions, observations and rewards are just more tokens to be modeled, and if I were running Google I would be busy instructing all work units to start packaging up such sequences of tokens to feed into the training runs for Gemini models. Many seemingly minor tasks (e.g. app recommendation in the Play store) either have, or could have, components of RL built into the pipeline, and could benefit from incorporating LLMs, either by putting the RL task in-context or through fine-tuning of very fast cheap models.
So when I say I don’t see a distinction between LLMs and “short term planning agents” I mean that we already know how to subsume RL tasks into next token prediction, and so there is in some technical sense already no distinction. It’s a question of how the underlying capabilities are packaged and deployed, and I think that within 6-12 months there will be many internal deployments of LLMs doing short sequences of tasks within Google. If that works, then it seems very natural to just scale up sequence length as generalisation improves.
Arguably fine-tuning a next-token predictor on action, observation, reward sequences, or doing it in-context, is inferior to using algorithms like PPO. However, the advantage of knowledge transfer from the rest of the next-token predictor’s data distribution may more than compensate for this on some short-term tasks.
I think o1 is a partial realization of your thesis, and the only reason it’s not more successful is because the compute used for GPT-o1 and GPT-4o were essentially the same:
https://www.lesswrong.com/posts/bhY5aE4MtwpGf3LCo/openai-o1
And yeah, the search part was actually quite good, if a bit modest in it’s gains.
As far as I can tell Strawberry is proving me right: it’s going beyond pre-training and scales inference—the obvious next step.
A lot of people said just scaling pre-trained transformers would scale to AGI. I think that’s silly and doesn’t make sense. But now you don’t have to believe me—you can just use OpenAIs latest model.
The next step is to do efficient long-horizon RL for data-sparse domains.
Strawberry working suggest that this might not be so hard. Don’t be fooled by the modest gains of Strawberry so far. This is a new paradigm that is heading us toward true AGI and superintelligence.
Yeah actually Alexander and I talked about that briefly this morning. I agree that the crux is “does this basic kind of thing work” and given that the answer appears to be “yes” we can confidently expect scale (in both pre-training and inference compute) to deliver significant gains.
I’d love to understand better how the RL training for CoT changes the representations learned during pre-training.
in my reading, Strawberry is showing that indeed scaling just pretraining transformers will *not* lead to AGI. The new paradigm is inference-scaling—the obvious next step is doing RL on long horizons and sparse data domains. I have been saying this ever since gpt-3 came out.
For the question of general intelligence imho the scaling is conceptually a red herring: any (general purpose) algorithm will do better when scaled. The key in my mind is the algorithm not the resource, just like I would say a child is generally intelligent while a pocket calculator is not even if the child can’t count to 20 yet. It’s about the meta-capability to learn not the capability.
As we spoke earlier—it was predictable that this was going to be the next step. It was likely it was going to work, but there was a hopeful world in which doing the obvious thing turned out to be harder. That hope has been dashed—it suggests longer horizons might be easy too. This means superintelligence within two years is not out of the question.
We have been shown that this search algorithm works, and we not yet have been shown that the other approaches don’t work.
Remember, technological development is disjunctive, and just because you’ve shown that 1 approach works, doesn’t mean that we have been shown that only that approach works.
Of course, people will absolutely try to scale this one up now that they found success, and I think that timelines have definitely been shortened, but remember that AI progress is closer to a disjunctive scenario than conjunctive scenario:
I agree with this quote below, but I wanted to point out the disjunctiveness of AI progress:
https://gwern.net/forking-path
strong disagree. i would be highly surprised if there were multiple essentially different algorithms to achieve general intelligence*.
I also agree with the Daniel Murfet’s quote. There is a difference between a disjunction before you see the data and a disjunction after you see the data. I agree AI development is disjunctive before you see the data—but in hindsight all the things that work are really minor variants on a single thing that works.
*of course “essentially different” is doing a lot of work here. some of the conceptual foundations of intelligence haven’t been worked out enough (or Vanessa has and I don’t understand it yet) for me to make a formal statement here.
Re different algorithms, I actually agree with both you and Daniel Murfet in that conditional on non-reversible computers, there is at most 1-3 algorithms to achieve intelligence that can scale arbitrarily large, and I’m closer to 1 than 3 here.
But once reversible computers/superconducting wires are allowed, all bets are off on how many algorithms are allowed, because you can have far, far more computation with far, far less waste heat leaving, and a lot of the design of computers is due to heat requirements.
Reversible computing and superconducting wires seem like hardware innovations. You are saying that this will actually materially change the nature of the algorithm you’d want to run?
I’d bet against. I’d be surprised if this was the case. As far as I can tell everything we have so seen so far points to a common simple core of general intelligence algorithm (basically an open-loop RL algorithm on top of a pre-trained transformers). I’d be surprised if there were materially different ways to do this. One of the main takeaways of the last decade of deep learning process is just how little architecture matters—it’s almost all data and compute (plus I claim one extra ingredient, open-loop RL that is efficient on long horizons and sparse data novel domains)
I don’t know for certain of course. If I look at theoretical CS though the universality of computation makes me skeptical of radically different algorithms.
I’m a bit confused by what you mean by “LLMs will not scale to AGI” in combination with “a single innovation is all that is needed for AGI”.
E.g., consider the following scenarios:
AGI (in the sense you mean) is achieved by figuring out a somewhat better RL scheme and massively scaling this up on GPT-6.
AGI is achieved by doing some sort of architectural hack on top of GPT-6 which makes it able to reason in neuralese for longer and then doing a bunch of training to teach the model to use this well.
AGI is achieved via doing some sort of iterative RL/synth data/self-improvement process for GPT-6 in which GPT-6 generates vast amounts of synthetic data for itself using various tools.
IMO, these sound very similar to “LLMs scale to AGI” for many practical purposes:
LLM scaling is required for AGI
LLM scaling drives the innovation required for AGI
From the public’s perspective, it maybe just looks like AI is driven by LLMs getting better over time and various tweaks might be continuously introduced.
Maybe it is really key in your view that the single innovation is really discontinuous and maybe the single innovation doesn’t really require LLM scaling.
I think a single innovation left to create LTPA is unlikely because it runs contrary to the history of technology and of machine learning. For example, in the 10 years before AlphaGo and before GPT-4, several different innovations were required—and that’s if you count “deep learning” as one item. ChatGPT actually understates the number here because different components of the transformer architecture like attention, residual streams, and transformer++ innovations were all developed separately.
I mostly regard LLMs = [scaling a feedforward network on large numbers of GPUs and data] as a single innovation.
Then I think you should specify that progress within this single innovation could be continuous over years and include 10+ ML papers in sequence each developing some sub-innovation.
Agreed on all points except a couple of the less consequential, where I don’t disagree.
Strongest agreement: we’re underestimating the importance of governments for alignment and use/misuse. We haven’t fully updated from the inattentive world hypothesis. Governments will notice the importance of AGI before it’s developed, and will seize control. They don’t need to nationalize the corporations, they just need to have a few people embedded at theh company and demand on threat of imprisonment that they’re kept involved with all consequential decisions on its use. I doubt they’d even need new laws, because the national security implications are enormous. But if they need new laws, they’ll create them as rapidly as necessary. Hopping borders will be difficult, and just put a different government in control.
Strongest disagreement: I think it’s likely that zero breakthroughs are needed to add long term planning capabilities to LLM-based systems, and so long term planning agents (I like the terminology) will be present very soon, and improve as LLMs continue to improve. I have specific reasons for thinking this. I could easily be wrong, but I’m pretty sure that the rational stance is “maybe”. This maybe advances the timelines dramatically.
Also strongly agree on AGI as a relatively discontinuous improvement; I worry that this is glossed over in modern “AI safety” discussions, causing people to mistake controlling LLMs for aligning the AGIs we’ll create on top of them. AGI alignment requires different conceptual work.
Do you think the final big advance happens within or with-out labs?
Probably within.
So somebody gets an agent which efficiently productively indefinitely works on any specified goal, then they just let the government find out and take it? No countermeasures?
What “coherent, long-term planning agents” means, and what is possible with these agents, is not clear to me. How would they overcome lack of access to knowledge, as was highlighted by F.A. Hayek in “The Use of Knowledge in Society”? What actions would they plan? How would their planning come to replace humans’ actions? (Achieving control over some sectors of battlefields would only be controlling destruction, of course, it would not be controlling creation.)
Some discussion is needed that recognizes and takes into account differences among governance structures. What seems the most relevant to me are these cases: (1) totalitarian governments, (2) somewhat-free governments, (3) transnational corporations, (4) decentralized initiatives. This is a new kind of competition, but the results will be like with major wars: Resilient-enough groups will survive the first wave or new groups will re-form later, and ultimately the competition will be won by the group that outproduces the others. In each successive era, the group that outproduces the others will be the group that leaves people the freest.