In general, I have noticed a pattern where people are dismissive of recursive self improvement. To the extent people are still believing this, I would like to suggest this is a cached thought that needs to be refreshed.
When it seemed like models with a chance of understanding code or mathematics were a long ways off—which it did (checks notes) two years ago, this may have seemed sane. I don’t think it seems sane anymore.
What would it look like to be on the precipice of a criticality threshold? I think it looks like increasingly capable models making large strides in coding and mathematics. I think it looks like feeding all of human scientific output into large language models. I think it looks a world where a bunch of corporations are throwing hundreds of millions of dollars into coding models and are now in the process of doing the obvious things that are obvious to everyone.
There’s a garbage article going around with rumors of GPT-4, which appears to be mostly wrong. But from slightly-more reliable rumors, I’ve heard it’s amazing and they’re picking the low-hanging data set optimization fruits.
The threshold for criticality, in my opinion, requires a model capable of understanding the code that produced it as well as a certain amount of scientific intuition and common sense. This no longer seems very far away to me.
It may also need a structured training environment and heuristic to select for generality.
The structured training environment is a set of tasks that train the machine a large breadth of base knowledge and skills to be a general AI.
The heuristic is just the point system : what metric are we selecting AI candidates by. Presumably we want a metric that selects simpler and smaller candidates with architecture that are heavily reused—something that looks like the topology of a brain—but maybe that won’t work.
So the explosion takes several things: compute, recursion, a software stack framework that is composable enough for automated design iteration, a bench, a heuristic.
Nukes weren’t really simple either, there were a lot of steps especially for the first implosion device. It took an immense amount of money and resources from the time physicists realized it was possible.
I think people are ignoring criticality because it hasn’t shown any gain in the history of ai because past systems were too simple. It’s not a proven track to success. What does work is bigass transformers.
I suppose I expect recursive self-improvement to play out in the course of months not years. And I worry groups like OpenAI are insane enough to pursue recursive self improvement as an explicit engineering goal. (Altman seems to be a moral realist, explicitly says he thinks the orthogonality thesis is false.) From the outside, it will appear instant as there will be a perceived discontinuity when the fact that it has achieved a decisive strategic advantage becomes obvious.
Well again remember a nuclear device is a critical mass of weapons grade material.
Anything less than weapons grade and nothing happens.
Anything less than sudden explosive combination of the materials and the device will heat itself up and blast itself apart with sub kiloton yield.
So analogy wise : current llms can “babble” out code that sometimes even works. They are not trained on RL selecting for correct and functional code.
Self improvement by code generation isn’t yet possible.
Other groups have tried making neural networks composable, and using one neural network based agent to design others. It is also not good enough for recursion but this is how autoML works.
Basically our enrichment isn’t high enough and so nothing will happen. The recursion quenches itself before it can start, the first generation output isn’t even functional.
But yes, at some future point in time it WILL be strong enough and crazy shit will happen. I mean think about the nuclear example: all those decades of discovering nuclear physics, fission, the chain reaction, building a nuclear reactor, purifying the plutonium...all that time and the interesting event happened in milliseconds.
I’m honestly not even sure whether this comment is in support of or against my disagreements.
I’m skeptical of the “recursive self-improvement leads to enormous gains in intelligence over days” story but I support the “more automation leads to faster R&D leads to more automation, etc.” story which is also a form of recursive self-improvement-just over the span of years rather than days.
Recursion is an empirical fact. It works numerous places. Computing itself that led to AI was recursive—the tools that allow for high end/high performance silicon chips require previous generations of those chips to function. (to be fair the coupling is loose, Intel could design better chips using 5-10 year old desktops and servers and some of their equipment is that outdated)
Electric power generator production for all currently used types require electric power from prior generators, etc.
I think making progress on ML is pretty hard. In order for a single AI to self improve quickly enough that it changed timelines, it would have to improve close to as fast as the speed at which all of the humans working on it could improve it. I don’t know why you would expect to see such superhuman coding/science capabilities without other kinds of superintelligence.
In general, I have noticed a pattern where people are dismissive of recursive self improvement. To the extent people are still believing this, I would like to suggest this is a cached thought that needs to be refreshed.
When it seemed like models with a chance of understanding code or mathematics were a long ways off—which it did (checks notes) two years ago, this may have seemed sane. I don’t think it seems sane anymore.
What would it look like to be on the precipice of a criticality threshold? I think it looks like increasingly capable models making large strides in coding and mathematics. I think it looks like feeding all of human scientific output into large language models. I think it looks a world where a bunch of corporations are throwing hundreds of millions of dollars into coding models and are now in the process of doing the obvious things that are obvious to everyone.
There’s a garbage article going around with rumors of GPT-4, which appears to be mostly wrong. But from slightly-more reliable rumors, I’ve heard it’s amazing and they’re picking the low-hanging data set optimization fruits.
The threshold for criticality, in my opinion, requires a model capable of understanding the code that produced it as well as a certain amount of scientific intuition and common sense. This no longer seems very far away to me.
But then, I’m no ML expert.
It may also need a structured training environment and heuristic to select for generality.
The structured training environment is a set of tasks that train the machine a large breadth of base knowledge and skills to be a general AI.
The heuristic is just the point system : what metric are we selecting AI candidates by. Presumably we want a metric that selects simpler and smaller candidates with architecture that are heavily reused—something that looks like the topology of a brain—but maybe that won’t work.
So the explosion takes several things: compute, recursion, a software stack framework that is composable enough for automated design iteration, a bench, a heuristic.
Nukes weren’t really simple either, there were a lot of steps especially for the first implosion device. It took an immense amount of money and resources from the time physicists realized it was possible.
I think people are ignoring criticality because it hasn’t shown any gain in the history of ai because past systems were too simple. It’s not a proven track to success. What does work is bigass transformers.
I suppose I expect recursive self-improvement to play out in the course of months not years. And I worry groups like OpenAI are insane enough to pursue recursive self improvement as an explicit engineering goal. (Altman seems to be a moral realist, explicitly says he thinks the orthogonality thesis is false.) From the outside, it will appear instant as there will be a perceived discontinuity when the fact that it has achieved a decisive strategic advantage becomes obvious.
Well again remember a nuclear device is a critical mass of weapons grade material.
Anything less than weapons grade and nothing happens.
Anything less than sudden explosive combination of the materials and the device will heat itself up and blast itself apart with sub kiloton yield.
So analogy wise : current llms can “babble” out code that sometimes even works. They are not trained on RL selecting for correct and functional code.
Self improvement by code generation isn’t yet possible.
Other groups have tried making neural networks composable, and using one neural network based agent to design others. It is also not good enough for recursion but this is how autoML works.
Basically our enrichment isn’t high enough and so nothing will happen. The recursion quenches itself before it can start, the first generation output isn’t even functional.
But yes, at some future point in time it WILL be strong enough and crazy shit will happen. I mean think about the nuclear example: all those decades of discovering nuclear physics, fission, the chain reaction, building a nuclear reactor, purifying the plutonium...all that time and the interesting event happened in milliseconds.
I’m honestly not even sure whether this comment is in support of or against my disagreements.
I’m skeptical of the “recursive self-improvement leads to enormous gains in intelligence over days” story but I support the “more automation leads to faster R&D leads to more automation, etc.” story which is also a form of recursive self-improvement-just over the span of years rather than days.
Recursion is an empirical fact. It works numerous places. Computing itself that led to AI was recursive—the tools that allow for high end/high performance silicon chips require previous generations of those chips to function. (to be fair the coupling is loose, Intel could design better chips using 5-10 year old desktops and servers and some of their equipment is that outdated)
Electric power generator production for all currently used types require electric power from prior generators, etc.
But yeah it so far hasn’t happened in days.
I think making progress on ML is pretty hard. In order for a single AI to self improve quickly enough that it changed timelines, it would have to improve close to as fast as the speed at which all of the humans working on it could improve it. I don’t know why you would expect to see such superhuman coding/science capabilities without other kinds of superintelligence.