Back in the day when this discourse first took off I remember meeting a lot of people that told me something in the vein of ‘Yudkowskys position that AI will just FOOM like that is silly, not credible, sci-fi. You should really be listening to this Paul guy. His continuous slow takeoff perspective is much more sensible’.
I remember pointing out to them that “Paul slow” according to Christiano was ‘like the industrial revolution but 100x faster’…
Depending on who you ask, the industrial revolution lasted ~150 years—so that’s 1,5 years. That’s pretty much what we are seeing. This was just status quo bias and respectability bias.
Actually, ofc the discourse was double silly. FOOM can still happen even in a ‘slow takeoff’ scenario. It’s a ‘slow’ straight line until some threshold is hit until it absolutely explodes. [although imho people equivocate between different versions of FOOM in confusing ways]
Whether or not a certain takeoff will happen only partially has to do with predictable features like scaling compute. Part of it may be predicted by straight-line numerology but a lot of it comes down to harder-to-predict or even impossible-to-predict features about the future like algorithmic & architectural changes, government intervention etc. That was the reasonable position in 2020 anyway. Today we have enough bits to have a pretty good guess when where and how superintelligence will happen.
Hi @Akash here are my current best guesses on the arrival and nature of superintelligence:
Timeframe probably around end of the decade 2028-2032 but possibly even earlier. In the unlikely but not impossible case that AGI has not been achieved by 2035 timelines lengthen again.
Players will be the usual suspects: openAi, Anthropic, deepMind, xAI, Meta, Baidu etc. Compute spend will be in hundreds of billions maybe more. The number of players will be exceedingly small; likely some or all of the labs will join forces.
Likely there will be soft nationalization of AGI under US government under pressure of ai safety concerns, national security and the economics of giant GPU clusters.
China will have joined the race to AGI, and will likely build AGI systems not too long after US (conditional on no Crazy Things happening in the interim).
There will likely be various international agreements that will be ignored to varying degree.
AGI will be similar in some respects to current day LLMs in having a pre-training phase on large internet data but importantly will differ in being a continually ‘open-loop’ trained RL agent on top of a LLM chassis. The key will be doing efficient long-horizon RL on thoughts on top of current day pretrained transformers.
An important difference between current day LLMs and future superintelligent AGI is that reactions to prompts (like solve cancer, the Riemann hypothesis, selfreplicating drones) can take many hours, weeks, months. So directing the AGI in what questions to think about will be plausibly very important.
______________________<
After AGI is achieved (in the US) the future seems murky (to me). Seems hard to forecast how the public and decision-making elites will react exactly.
Likely the US government will try to control its use to a significant degree. It’s very plausible that the best AI will not be accessible fully to the public.
National security interests likely to become increasingly important.
A lot of compute likely to go into making AGI think about health &medicine.
Other countries (chief among them China) will want to build their own.
Very large and powerful language models will still be available to the public.
Yeah, one of OpenAIs goals is to make AI models think for very long and get better answers the more they think, limited only by available computing power, so this will almost certainly be an important subgoal to this:
An important difference between current day LLMs and future superintelligent AGI is that reactions to prompts (like solve cancer, the Riemann hypothesis, selfreplicating drones) can take many hours, weeks, months
And o1, while not totally like the system proposed, is a big conceptual advance to making this a reality.
Actually, much more extreme than the industrial revolution because at the end of the singularity you’ll probably discover a huge fraction of all technology and be able to quickly multiply energy production by a billion.
That said, I think Paul expects a timeframe longer than 1.5 years, at least he did historically, maybe this has changed with updates over the last few years. (Perhaps more like 5-10 years.)
Back in the day when this discourse first took off I remember meeting a lot of people that told me something in the vein of ‘Yudkowskys position that AI will just FOOM like that is silly, not credible, sci-fi. You should really be listening to this Paul guy. His continuous slow takeoff perspective is much more sensible’.
I remember pointing out to them that “Paul slow” according to Christiano was ‘like the industrial revolution but 100x faster’…
Depending on who you ask, the industrial revolution lasted ~150 years—so that’s 1,5 years. That’s pretty much what we are seeing. This was just status quo bias and respectability bias.
Actually, ofc the discourse was double silly. FOOM can still happen even in a ‘slow takeoff’ scenario. It’s a ‘slow’ straight line until some threshold is hit until it absolutely explodes. [although imho people equivocate between different versions of FOOM in confusing ways]
Whether or not a certain takeoff will happen only partially has to do with predictable features like scaling compute. Part of it may be predicted by straight-line numerology but a lot of it comes down to harder-to-predict or even impossible-to-predict features about the future like algorithmic & architectural changes, government intervention etc. That was the reasonable position in 2020 anyway. Today we have enough bits to have a pretty good guess when where and how superintelligence will happen.
@Alexander Gietelink Oldenziel can you expand on this? What’s your current model of when/where/how superintelligence will happen?
Hi @Akash here are my current best guesses on the arrival and nature of superintelligence:
Timeframe probably around end of the decade 2028-2032 but possibly even earlier. In the unlikely but not impossible case that AGI has not been achieved by 2035 timelines lengthen again.
Players will be the usual suspects: openAi, Anthropic, deepMind, xAI, Meta, Baidu etc. Compute spend will be in hundreds of billions maybe more. The number of players will be exceedingly small; likely some or all of the labs will join forces.
Likely there will be soft nationalization of AGI under US government under pressure of ai safety concerns, national security and the economics of giant GPU clusters. China will have joined the race to AGI, and will likely build AGI systems not too long after US (conditional on no Crazy Things happening in the interim). There will likely be various international agreements that will be ignored to varying degree.
AGI will be similar in some respects to current day LLMs in having a pre-training phase on large internet data but importantly will differ in being a continually ‘open-loop’ trained RL agent on top of a LLM chassis. The key will be doing efficient long-horizon RL on thoughts on top of current day pretrained transformers.
An important difference between current day LLMs and future superintelligent AGI is that reactions to prompts (like solve cancer, the Riemann hypothesis, selfreplicating drones) can take many hours, weeks, months. So directing the AGI in what questions to think about will be plausibly very important.
______________________< After AGI is achieved (in the US) the future seems murky (to me). Seems hard to forecast how the public and decision-making elites will react exactly.
Likely the US government will try to control its use to a significant degree. It’s very plausible that the best AI will not be accessible fully to the public.
National security interests likely to become increasingly important. A lot of compute likely to go into making AGI think about health &medicine.
Other countries (chief among them China) will want to build their own. Very large and powerful language models will still be available to the public.
Yeah, one of OpenAIs goals is to make AI models think for very long and get better answers the more they think, limited only by available computing power, so this will almost certainly be an important subgoal to this:
And o1, while not totally like the system proposed, is a big conceptual advance to making this a reality.
Just pointing out that you should transition to LessWrong Docs when trying to call out someone by tagging them.
I’ll do that right now:
@Akash
Response to Alexander Oldenziel’s models?
Actually, much more extreme than the industrial revolution because at the end of the singularity you’ll probably discover a huge fraction of all technology and be able to quickly multiply energy production by a billion.
That said, I think Paul expects a timeframe longer than 1.5 years, at least he did historically, maybe this has changed with updates over the last few years. (Perhaps more like 5-10 years.)