That’s the crux of this scenario, whether current AIs with near future improvements can do research. If they can, with scaling they only do it better. If they can’t, scaling might fail to help, even if they become agentic and therefore start generating serious money. That’s the sense in which AIs capable of 10 hours of work don’t lead to game-changing acceleration of research, by remaining incapable of some types of work.
What seems inevitable at the moment is AIs gainingworldmodels where they can reference any concepts that frequently come up in the training data. This promises proficiency in arbitrary routine tasks, but not necessarily construction of novel ideas that lack sufficient footprint in the datasets. Ability to understand such ideas in-context when explained seems to be increasing with LLM scale though, and might be crucial for situational awareness needed for becoming agentic, as every situation is individually novel.
That’s the crux of this scenario, whether current AIs with near future improvements can do research. If they can, with scaling they only do it better. If they can’t, scaling might fail to help, even if they become agentic and therefore start generating serious money. That’s the sense in which AIs capable of 10 hours of work don’t lead to game-changing acceleration of research, by remaining incapable of some types of work.
What seems inevitable at the moment is AIs gaining world models where they can reference any concepts that frequently come up in the training data. This promises proficiency in arbitrary routine tasks, but not necessarily construction of novel ideas that lack sufficient footprint in the datasets. Ability to understand such ideas in-context when explained seems to be increasing with LLM scale though, and might be crucial for situational awareness needed for becoming agentic, as every situation is individually novel.