I’m seeing a lot of people on LW saying that they have very short timelines (say, five years or less) until AGI. However, the arguments that I’ve seen often seem to be just one of the following:
“I’m not going to explain but I’ve thought about this a lot”
“People at companies like OpenAI, Anthropic etc. seem to believe this”
“Feels intuitive based on the progress we’ve made so far”
At the same time, it seems like this is not the majority view among ML researchers. The most recent representative expert survey that I’m aware of is the 2023 Expert Survey on Progress in AI. It surveyed 2,778 AI researchers who had published peer-reviewed research in the prior year in six top AI venues (NeurIPS, ICML, ICLR, AAAI, IJCAI, JMLR); the median time for a 50% chance of AGI was either in 23 or 92 years, depending on how the question was phrased.
While it has been a year since fall 2023 when this survey was conducted, my anecdotal impression is that many researchers not in the rationalist sphere still have significantly longer timelines, or do not believe that current methods would scale to AGI.
A more recent, though less broadly representative, survey is reported in Feng et al. 2024, In the ICLR 2024 “How Far Are We From AGI” workshop, 138 researchers were polled on their view. “5 years or less” was again a clear minority position, with 16.6% respondents. On the other hand, “20+ years” was the view held by 37% of the respondents.
Most recently, there were a number of “oh AGI does really seem close” comments with the release of o3. I mostly haven’t seen these give very much of an actual model for their view either; they seem to mostly be of the “feels intuitive” type. There have been some posts discussing the extent to which we can continue to harness compute and data for training bigger models, but that says little about the ultimate limits of the current models.
The one argument that I did see that felt somewhat convincing were the “data wall” and “unhobbling” sections of the “From GPT-4 to AGI” chapter of Leopold Aschenbrenner’s “Situational Awareness”, that outlined ways in which we could build on top of the current paradigm. However, this too was limited to just “here are more things that we could do”.
So, what are the strongest arguments for AGI being very close? I would be particularly interested in any discussions that explicitly look at the limitations of the current models and discuss how exactly people expect those to be overcome.
Here’s the structure of the argument that I am most compelled by (I call it the benchmarks + gaps argument), I’m uncertain about the details.
Focus on the endpoint of substantially speeding up AI R&D / automating research engineering. Let’s define our timelines endpoint as something that ~5xs the rate of AI R&D algorithmic progress (compared to a counterfactual world with no post-2024 AIs). Then make an argument that ~fully automating research engineering (experiment implementation/monitoring) would do this, along with research taste of at least the 50th percentile AGI company researcher (experiment ideation/selection).
Focus on REBench since it’s the most relevant benchmark. REBench is the most relevant benchmark here, for simplicity I’ll focus on only this though for robustness more benchmarks should be considered.
Based on trend extrapolation and benchmark base rates, roughly 50% we’ll saturate REBench by end of 2025.
Identify the most important gaps between saturating REBench and the endpoint defined in (1). The most important gaps between saturating REBench and achieving the 5xing AI R&D algorithmic progress are: (a) time horizon as measured by human time spent (b) tasks with worse feedback loops (c) tasks with large codebases (d) becoming significantly cheaper and/or faster than humans. There are some more more but they probably aren’t as important, should also take into account unknown gaps.
When forecasting the time to cross the gaps, it seems quite plausible that we get to the substantial AI R&D speedup within a few years after saturating REBench, so by end of 2028 (and significantly earlier doesn’t seem crazy).
This is the most important part of the argument, and one that I have lots of uncertainty over. We have some data regarding the “crossing speed” of some of the gaps but the data are quite limited at the moment. So there are a lot of judgment calls needed and people with strong long timelines intuitions might think the remaining gaps will take a long time to cross without this being close to falsified by our data.
This is broken down into “time to cross the gaps at 2024 pace of progress” → adjusting based on compute forecasts and intermediate AI R&D speedups before reaching 5x.
From substantial AI R&D speedup to AGI. Once we have the 5xing AIs, that’s potentially already AGI by some definitions but if you have a stronger one, the possibility of a somewhat fast takeoff means you might get it within a year or so after.
One reason I like this argument is that it will get much stronger over time as we get more difficult benchmarks and otherwise get more data about how quickly the gaps are being crossed.
I have a longer draft which makes this argument but it’s quite messy and incomplete and might not add much on top of the above summary for now. Unfortunately I’m prioritizing other workstreams over finishing this at the moment. DM me if you’d really like a link to the messy draft.
RE-bench tasks (see page 7 here) are not the kind of AI research where you’re developing new AI paradigms and concepts. The tasks are much more straightforward than that. So your argument is basically assuming without argument that we can get to AGI with just the more straightforward stuff, as opposed to new AI paradigms and concepts.
If we do need new AI paradigms and concepts to get to AGI, then there would be a chicken-and-egg problem in automating AI research. Or more specifically, there would be two categories of AI R&D, with the less important R&D category (e.g. performance optimization and other REbench-type tasks) being automatable by near-future AIs, and the more important R&D category (developing new AI paradigms and concepts) not being automatable.
(Obviously you’re entitled to argue / believe that we don’t need need new AI paradigms and concepts to get to AGI! It’s a topic where I think reasonable people disagree. I’m just suggesting that it’s a necessary assumption for your argument to hang together, right?)
I disagree. I think the existing body of published computer science and neuroscience research are chock full of loose threads. Tons of potential innovations just waiting to be harvested by automated researchers. I’ve mentioned this idea elsewhere. I call it an ‘innovation overhang’. Simply testing interpolations and extrapolations (e.g. scaling up old forgotten ideas on modern hardware) seems highly likely to reveal plenty of successful new concepts, even if the hit rate per attempt is low. I think this means a better benchmark would consist of: taking two existing papers, finding a plausible hypothesis which combines the assumptions from the papers, designs and codes and runs tests, then reports on results.
So I don’t think “no new concepts” is a necessary assumption for getting to AGI quickly with the help of automated researchers.
Is this bottlenecked by programmer time or by compute cost?
Both? If you increase only one of the two the other becomes the bottleneck?
I agree this means that the decision to devote substantial compute to both inference and to assigning compute resources for running experiments designed by AI reseachers is a large cost. Presumably, as the competence of the AI reseachers gets higher, it feels easier to trust them not to waste their assigned experiment compute.
There was discussion on Dwarkesh Patel’s interview with researcher friends where there was mention that AI reseachers are already restricted by compute granted to them for experiments. Probably also on work hours per week they are allowed to spend on novel “off the main path” research.
So in order for there to be a big surge in AI R&D there’d need to be prioritization of that at a high level. This would be a change of direction from focusing primarily on scaling current techniques rapidly, and putting out slightly better products ASAP.
So yes, if you think that this priority shift won’t happen, then you should doubt that the increase in R&D speed my model predicts will occur.
But what would that world look like? Probably a world where scaling continues to pay dividends, and getting to AGI is more straightforward yhan Steve Byrnes or I expect.
I agree that that’s a substantial probability, but it’s also an AGI-soon sort of world.
I argue that for AGI to be not-soon, you need both scaling to fail and for algorithm research to fail.
It sounds like your disagreement isn’t with drawing a link from RE-bench to forecasting when we can automate research engineering, but is instead with thinking that you can get AGI shortly after automating research engineering due to AI R&D acceleration and already being pretty close. Is that right?
Note that the comment says research engineering, not research scientists.
I’ve been arguing for 2027-ish AGI for several years now. I do somewhat fall into the annoying category of refusing to give my full details for believing this (publicly). I’ve had some more in-depth discussions about this privately.
One argument I have been making publicly is that I think Ajeya’s Bioanchors report greatly overestimated human brain compute. I think a more careful reading of Joe Carlsmith’s report that hers was based on supports my own estimates of around 1e15 FLOPs.
Connor Leahy makes some points I agree with in his recent Future of Life interview. https://futureoflife.org/podcast/connor-leahy-on-why-humanity-risks-extinction-from-agi/
Another very relevant point is that recent research on the human connectome shows that long-range connections (particularly between regions of the cortex) are lower bandwidth than was previously thought. Examining this bandwidth in detail leads me to believe that efficient decentralized training should be possible. Even with considering that training a human brain equivalent model would require 10000x parallel brain equivalents to have a reasonable training time, the current levels of internet bandwidth between datacenters worldwide should be more than sufficient.
Thus, my beliefs are strongly pointint towards: “with the right algorithms we will have more than good enough hardware and more than sufficient data. Also, those algorithms are available to be found, and are hinted at by existing neuroscience data.” Thus, with AI R&D accelerated research on algorithms, we should expect rapid progress on peak capabilities and efficiency which doesn’t plateau at human-peak-capability or human-operation-speed. Super-fast and super-smart AGI within a few months of full AGI, and rapidly increasing speeds of progress leading up to AGI.
If I’m correct, then the period of time from 2026 to 2027 will contain as much progress on generally intelligent systems as all of history leading up to 2026. ASI will thus be possible before 2028.
Only social factors (e.g. massively destructive war or unprecedented international collaboration on enforcing an AI pause) will change these timelines.
Further thoughts here: A path to human autonomy
Lots of disagree votes, but no discussion. So annoying when that happens.
Propose a bet! Ask for my sources! Point out a flaw in my reasoning! Don’t just disagree and walk away!
I feel like a bet is fundamentally unfair here because in the cases where I’m wrong, there’s a high chance that I’ll be dead anyway and don’t have to pay. The combination of long timelines but high P(doom|AGI soon) means I’m not really risking my reputation/money in the way I’m supposed to with a bet. Are you optimistic about alignment, or does this asymmetry not bother you for other reasons? (And I don’t have the money to make a big bet regardless.)
Great question! Short answer: I’m optimistic about muddling through with partial alignment combined with AI control and AI governance (limiting peak AI capabilities, global enforcement of anti-rogue-AI, anti-self-improving-AI, and anti-self-replicating-weapons laws). See my post “A Path to Human Autonomy” for more details.
I also don’t have money for big bets. I’m more interested in mostly-reputation-wagers about the very near future. So that I might get my reputational returns in time for them to pay off in respectful-attention-from-powerful-decisionmakers, which in turn I would hope might pay off in better outcomes for me, my loved ones, and humanity.
If I am incorrect, then I want to not be given the ear of decision makers, and I want them to instead pay more attention to someone with better models than me. Thus, seems to me like a fairly win-win situation to be making short term reputational bets.
Gotcha. I’m happy to offer 600 of my reputation points vs. 200 of yours on your description of 2026-2028 not panning out. (In general if it becomes obvious[1] that we’re racing toward ASI in the next few years, then people should probably not take me seriously anymore.)
well, so obvious that I agree, anyway; apparently it’s already obvious to some people.
I’ll happily accept that bet, but maybe we could also come up with something more specific about the next 12 months?
Example: https://manifold.markets/MaxHarms/will-ai-be-recursively-self-improvi
Not that one; I would not be shocked if this market resolves Yes. I don’t have an alternative operationalization on hand; would have to be about AI doing serious intellectual work on real problems without any human input. (My model permits AI to be very useful in assisting humans.)
Can we bet karma?
Edit: sarcasm
Feeding this norm creates friction, filters evidence elicited in the agreement-voting. If there is a sense that a vote needs to be explained, it often won’t be cast.
Agree. I do think it is annoying, but allowing people to do that is quite crucial for the integrity of the voting system.
I think the algorithm progress is doing some heavy lifting in this model. I think if we had a future textbook on agi we could probably build one but AI is kinda famous for minor and simple things just not being implemented despite all the parts being there
See ReLU activations and sigmoid activations.
If we’re bottlenecking at algorithms alone is there a reason that isn’t a really bad bottleneck?
See my other response to Raphael elsewhere in this comment thread.
My model is that the big AI labs are currently throttling algorithmic progress by choosing to devote their resources to scaling.
If scaling leads to AGI, we get AGI soon that way. (I give this about 20% chance.)
If scaling doesn’t lead to AGI, then refocusing resources on experimentation seems like a natural next move. (I think this is about 80% likely to work in under two years if made a major focus of resources, including both giving human researchers the time, encouragement and compute resources they need, plus developing increasingly helpful AI reseachers.)
Hmm, mixed agree/disagree. Scale probably won’t work, algorithms probably would, but I don’t think it’s going to be that quick.
Namely, I think that the company struggling with fixed capital costs could accomplish much more, much quicker using the salary expenses of the top researchers they already have they’d have done it or gave it a good try at least
I’m 5 percent that a serious switch to algorithms would result in AGI in 2 years. You might be more well read than me on this so I’m not quite taking side bets right now!
An AGI broadly useful for humans needs to be good at general tasks for which currently there is no way of finding legible problem statements (where System 2 reasoning is useful) with verifiable solutions. Currently LLMs are slightly capable at such tasks, and there are two main ways in which they become more capable, scaling and RL.
Scaling is going to continue rapidly showing new results at least until 2026-2027, probably also 2028-2029. If there’s no AGI or something like a $10 trillion AI company by then, there won’t be a trillion dollar training system and the scaling experiments will fall back to the rate of semiconductor improvement.
Then there’s RL, which as o3 demonstrates applies to LLMs as a way of making them stronger and not merely eliciting capabilities formed in pretraining. But it only works directly around problem statements with verifiable solutions, and it’s unclear how to generate them for more general tasks or how far will the capabilities generalize from the training problems that are possible to construct in bulk. (Arguably self-supervised learning is good at instilling general capabilities because the task of token prediction is very general, it subsumes all sorts of things. But it’s not legible.) Here too scale might help with generalization stretching further from the training problems, and with building verifiable problem statements for more general tasks, and we won’t know how much it will help until the experiments are done.
So my timelines are concentrated on 2025-2029, after that the rate of change in capabilities goes down. Probably 10 more years of semiconductor and algorithmic progress after that are sufficient to wrap it up though, so 2040 without AGI seems unlikely.
I think the gaps between where we are and human-level (and broadly but not precisely human-like) cognition are smaller than they appear. Modest improvements in to-date neglected cognitive systems can allow LLMs to apply their cognitive abilities in more ways, allowing more human-like routes to performance and learning. These strengths will build on each other nonlinearly (while likely also encountering unexpected roadblocks).
Timelines are thus very difficult to predict, but ruling out very short timelines based on averaging predictions without gears-level models of fast routes to AGI would be a big mistake. Whether and how quickly they work is an empirical question.
One blocker to taking short timelines seriously is the belief that fast timelines mean likely human extinction. I think they’re extremely dangerous but that possible routes to alignment also exist—but that’s a separate question.
I also think this is the current default path, or I wouldn’t describe it.
I think my research career using deep nets and cognitive architectures to understand human cognition is pretty relevant for making good predictions on this path to AGI. But I’m biased, just like everyone else.
Anyway, here’s very roughly why I think the gaps are smaller than they appear.
Current LLMs are like humans with excellent:
language abilities,
semantic memory
working memory
They can now do almost all short time-horizon tasks that are framed in language better than humans. And other networks can translate real-world systems into language and code, where humans haven’t already done it.
But current LLMs/foundation models are dramatically missing some human cognitive abilities:
Almost no episodic memory for specific important experiences
No agency—they do only what they’re told
Poor executive function (self-management of cognitive tasks)
Relatedly, bad/incompetent at long time-horizon tasks.
And zero continuous learning (and self-directed learning)
Crucial for human performance on complex tasks
Those lacks would appear to imply long timelines.
But both long time-horizon tasks and self-directed learning are fairly easy to reach. The gaps are not as large as they appear.
Agency is as simple as repeatedly calling a prompt of “act as an agent working toward goal X; use tools Y to gather information and take actions as appropriate”. The gap between a good oracle and an effective agent is almost completely illusory.
Episodic memory is less trivial, but still relatively easy to improve from current near-zero-effort systems. Efforts from here will likely build on LLMs strengths. I’ll say no more publicly; DM me for details. But it doesn’t take a PhD in computational neuroscience to rederive this, which is the only reason I’m mentioning it publicly. More on infohazards later.
Now to the capabilities payoff: long time-horizon tasks and continuous, self-directed learning.
Long time-horizon task abilities are an emergent product of episodic memory and general cognitive abilities. LLMs are “smart” enough to manage their own thinking; they don’t have instructions or skills to do it. o1 appears to have those skills (although no episodic memory which is very helpful in managing multiple chains of thought), so similar RL training on Chains of Thought is probably one route achieving those.
Humans do not mostly perform long time-horizon tasks by trying them over and over. They either ask someone how to do it, then memorize and reference those strategies with episodic memory; or they perform self-directed learning, and pose questions and form theories to answer those same questions.
Humans do not have or need “9s of reliability” to perform long time-horizon tasks. We substitute frequent error-checking and error-correction. We then learn continuously on both strategy (largely episodic memory) and skills/habitual learning (fine-tuning LLMs already provides a form of this habitization of explicit knowledge to fast implicit skills).
Continuous, self-directed learning is a product of having any type of new learning (memory), and using some of the network/agents’ cognitive abilities to decide what’s worth learning. This learning could be selective fine-tuning (like o1s “deliberative alignment), episodic memory, or even very long context with good access as a first step. This is how humans master new tasks, along with taking instruction wisely. This would be very helpful for mastering economically viable tasks, so I expect real efforts put into mastering it.
Self-directed learning would also be critical for an autonomous agent to accomplish entirely novel tasks, like taking over the world.
This is why I expect “Real AGI” that’s agentic and learns on its own, and not just transformative tool “AGI” within the next five years (or less). It’s easy and useful, and perhaps the shortest path to capabilities (as with humans teaching themselves).
If that happens, I don’t think we’re necessarily doomed, even without much new progress on alignment (although we would definitely improve our odds!). We are already teaching LLMs mostly to answer questions correctly and to follow instructions. As long as nobody gives their agent an open-ended top-level goal like “make me lots of money”, we might be okay. Instruction-following AGI is easier and more likely than value aligned AGI although I need to work through and clarify why I find this so central. I’d love help.
Convincing predictions are also blueprints for progress. Thus, I have been hesitant to say all of that clearly.
I said some of this at more length in Capabilities and alignment of LLM cognitive architectures and elsewhere. But I didn’t publish it in my previous neuroscience career nor have I elaborated since then.
But I’m increasingly convinced that all of this stuff is going to quickly become obvious to any team that sits down and starts thinking seriously about how to get from where we are to really useful capabilities. And more talented teams are steadily doing just that.
I now think it’s more important that the alignment community takes short timelines more seriously, rather than hiding our knowledge in hopes that it won’t be quickly rederived. There are more and more smart and creative people working directly toward AGI. We should not bet on their incompetence.
There could certainly be unexpected theoretical obstacles. There will certainly be practical obstacles. But even with expected discounts for human foibles and idiocy and unexpected hurdles, timelines are not long. We should not assume that any breakthroughs are necessary, or that we have spare time to solve alignment adequately to survive.
It’s worthy of a (long) post, but I’ll try to summarize. For what it’s worth, I’ll die on this hill.
General intelligence = Broad, cross-domain ability and skills.
Narrow intelligence = Domain-specific or task-specific skills.
The first subsumes the second at some capability threshold.
My bare bones definition of intelligence: prediction. It must be able to consistently predict itself & the environment. To that end it necessarily develops/evolves abilities like learning, environment/self sensing, modeling, memory, salience, planning, heuristics, skills, etc. Roughly what Ilya says about token prediction necessitating good-enough models to actually be able to predict that next token (although we’d really differ on various details)
Firstly, it’s based on my practical and theoretical knowledge of AI and insights I believe to have had into the nature of intelligence and generality for a long time. It also includes systems, cybernetics, physics, etc. I believe a holistic view helps inform best w.r.t. AGI timelines. And these are supported by many cutting edge AI/robotics results of the last 5-9 years (some old work can be seen in new light) and also especially, obviously, the last 2 or so.
Here are some points/beliefs/convictions I have for thinking AGI for even the most creative goalpost movers is basically 100% likely before 2030, and very likely much sooner. A fast takeoff also, understood as the idea that beyond a certain capability threshold for self-improvement, AI will develop faster than natural, unaugmented humans can keep up with.
It would be quite a lot of work to make this very formal, so here are some key points put informally:
- Weak generalization has been already achieved. This is something we are piggybacking off of already, and there is meaningful utility since GPT-3 or so. This is an accelerating factor.
- Underlying techniques (transformers , etc) generalize and scale.
- Generalization and performance across unseen tasks improves with multi-modality.
- Generalist models outdo specialist ones in all sorts of scenarios and cases.
- Synthetic data doesn’t necessarily lead to model collapse and can even be better than real world data.
- Intelligence can basically be brute-forced it looks like, so one should take Kurzweil *very* seriously (he tightly couples his predictions to increase in computation).
- Timelines shrunk massively across the board for virtually all top AI names/experts in the last 2 years. Top Experts were surprised by the last 2 years.
- Bitter Lesson 2.0.: there are more bitter lessons than Sutton’s, which are that all sorts of old techniques can be combined for great increases in results. See the evidence in papers linked below.
- “AGI” went from a taboo “bullshit pursuit for crackpots”, to a serious target of all major labs, publicly discussed. This means a massive increase in collective effort, talent, thought, etc. No more suppression of cross-pollination of ideas, collaboration, effort, funding, etc.
- The spending for AI only bolsters, extremely so, the previous point. Even if we can’t speak of a Manhattan Project analogue, you can say that’s pretty much what’s going on. Insane concentrations of talent hyper focused on AGI. Unprecedented human cycles dedicated to AGI.
- Regular software engineers can achieve better results or utility by orchestrating current models and augmenting them with simple techniques(RAG, etc). Meaning? Trivial augmentations to current models increase capabilities—this low hanging fruit implies medium and high hanging fruit (which we know is there, see other points).
I’d also like to add that I think intelligence is multi-realizable, and generality will be considered much less remarkable soon after we hit it and realize this than some still think it is.
Anywhere you look: the spending, the cognitive effort, the (very recent) results, the utility, the techniques...it all points to short timelines.
In terms of AI papers, I have 50 references or so I think support the above as well. Here are a few:
SDS : See it. Do it. Sorted Quadruped Skill Synthesis from Single Video Demonstration, Jeffrey L., Maria S., et al. (2024).
DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learning, Zhenyu J., Yuqi X., et in. (2024).
One-Shot Imitation Learning, Duan, Andrychowicz, et al. (2017).
Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks, Finn et al., (2017).
Unsupervised Learning of Semantic Representations, Mikolov et al., (2013).
A Survey on Transfer Learning, Pan and Yang, (2009).
Zero-Shot Learning—A Comprehensive Evaluation of the Good, the Bad and the Ugly, Xian et al., (2018).
Learning Transferable Visual Models From Natural Language Supervision, Radford et al., (2021).
Multimodal Machine Learning: A Survey and Taxonomy, Baltrušaitis et al., (2018).
Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine, Harsha N., Yin Tat Lee et al. (2023).
A Vision-Language-Action Flow Model for General Robot Control, Kevin B., Noah B., et al. (2024).
Open X-Embodiment: Robotic Learning Datasets and RT-X Models, Open X-Embodiment Collaboration, Abby O., et al. (2023).