If AIs of the near future can’t do good research (and instead are only proficient in concepts that have significant presence in datasets), singularity remains bottlenecked by human research speed. The way such AIs speed things up is through their commercial success making more investment in scaling possible, not directly (and there is little use for them in the lab). It’s currently unknown if scaling even at $1 trillion level is sufficient by itself, so some years of Futurama don’t seem impossible, especially as we are only talking 2029.
I think that AIs will be able to do 10 hours of research (at the level of a software engineer that gets paid $100k a year) within 4 years with 50% probability.
If we look at current systems, there’s not much indication that AI agents will be superhuman in non-AI-research tasks and subhuman in AI research tasks. One of the most productive uses of AI so far has been in helping software engineers code better, so I’d wager AI assistants will be even more helpful for AI research than for other things (compared to some prior based on those task’s “inherent difficulties”). Additionally, AI agents can do some basic coding using proper codebases and projects, so I think scaffolded GPT-5 or GPT-6 will likely be able to do much more than GPT-4.
That’s the crux of this scenario, whether current AIs with near future improvements can do research. If they can, with scaling they only do it better. If they can’t, scaling might fail to help, even if they become agentic and therefore start generating serious money. That’s the sense in which AIs capable of 10 hours of work don’t lead to game-changing acceleration of research, by remaining incapable of some types of work.
What seems inevitable at the moment is AIs gainingworldmodels where they can reference any concepts that frequently come up in the training data. This promises proficiency in arbitrary routine tasks, but not necessarily construction of novel ideas that lack sufficient footprint in the datasets. Ability to understand such ideas in-context when explained seems to be increasing with LLM scale though, and might be crucial for situational awareness needed for becoming agentic, as every situation is individually novel.
Note that AI doesn’t need to come up with original research ideas or do much original thinking to speed up research by a bunch. Even if it speeds up the menial labor of writing code, running experiments, and doing basic data analysis at scale, if you free up 80% of your researchers’ time, your researchers can now spend all of their time doing the important task, which means overall cognitive labor is 5x faster. This is ignoring effects from using your excess schlep-labor to trade against non-schlep labor leading to even greater gains in efficiency.
If AIs of the near future can’t do good research (and instead are only proficient in concepts that have significant presence in datasets), singularity remains bottlenecked by human research speed. The way such AIs speed things up is through their commercial success making more investment in scaling possible, not directly (and there is little use for them in the lab). It’s currently unknown if scaling even at $1 trillion level is sufficient by itself, so some years of Futurama don’t seem impossible, especially as we are only talking 2029.
I think that AIs will be able to do 10 hours of research (at the level of a software engineer that gets paid $100k a year) within 4 years with 50% probability.
If we look at current systems, there’s not much indication that AI agents will be superhuman in non-AI-research tasks and subhuman in AI research tasks. One of the most productive uses of AI so far has been in helping software engineers code better, so I’d wager AI assistants will be even more helpful for AI research than for other things (compared to some prior based on those task’s “inherent difficulties”). Additionally, AI agents can do some basic coding using proper codebases and projects, so I think scaffolded GPT-5 or GPT-6 will likely be able to do much more than GPT-4.
That’s the crux of this scenario, whether current AIs with near future improvements can do research. If they can, with scaling they only do it better. If they can’t, scaling might fail to help, even if they become agentic and therefore start generating serious money. That’s the sense in which AIs capable of 10 hours of work don’t lead to game-changing acceleration of research, by remaining incapable of some types of work.
What seems inevitable at the moment is AIs gaining world models where they can reference any concepts that frequently come up in the training data. This promises proficiency in arbitrary routine tasks, but not necessarily construction of novel ideas that lack sufficient footprint in the datasets. Ability to understand such ideas in-context when explained seems to be increasing with LLM scale though, and might be crucial for situational awareness needed for becoming agentic, as every situation is individually novel.
Note that AI doesn’t need to come up with original research ideas or do much original thinking to speed up research by a bunch. Even if it speeds up the menial labor of writing code, running experiments, and doing basic data analysis at scale, if you free up 80% of your researchers’ time, your researchers can now spend all of their time doing the important task, which means overall cognitive labor is 5x faster. This is ignoring effects from using your excess schlep-labor to trade against non-schlep labor leading to even greater gains in efficiency.