The above numbers suggest that (as long as sample efficiency doesn’t significantly improve) the world will always have enough compute to produce at least 23 million token-equivalents per second from any model that the world can afford to train (end-to-end, chinchilla-style). Notably, these are many more token-equivalents per second than we currently have human-AI-researcher-seconds per second. (And the AIs would have the further advantage of having much faster serial speeds.)
So once an AI system trained end-to-end can produce similarly much value per token as a human researcher can produce per second, AI research will be more than fully automated. This means that, when AI first contributes more to AI research than humans do, the average research progress produced by 1 token of output will be significantly less than an average human AI researcher produces in a second of thinking.
There’s probably a very similarly-shaped argument to be made based on difference in cost per token: because LLMs are much cheaper per token, the first time an LLM is as cost-efficient at producing AI research as a human researcher, it should be using many more tokens in its outputs (‘the average research progress produced by 1 token of output will be significantly less than an average human AI researcher produces in 1 token of output’). Which, similarly, should be helpful because ‘the token-by-token output of a single AI system should be quite easy for humans to supervise and monitor for danger’.
This framing might be more relevant from the POV of economic incentives to automate AI research (and I’m particularly interested in the analogous incentives to/feasibility of automating AI safety research).
There’s probably a very similarly-shaped argument to be made based on difference in cost per token: because LLMs are much cheaper per token, the first time an LLM is as cost-efficient at producing AI research as a human researcher, it should be using many more tokens in its outputs (‘the average research progress produced by 1 token of output will be significantly less than an average human AI researcher produces in 1 token of output’). Which, similarly, should be helpful because ‘the token-by-token output of a single AI system should be quite easy for humans to supervise and monitor for danger’.
This framing might be more relevant from the POV of economic incentives to automate AI research (and I’m particularly interested in the analogous incentives to/feasibility of automating AI safety research).