I think you should reexamine what I said by convergence. Do you...really...think a world that knows how to build (safe, usable tool) ASI would ever be stable by not building it. We are very close to that world, the time is measured in years if not months. Note that any party that gets it working long enough escapes the grim game, they can do whatever they want limited by physics. I acknowledge your point about chip production, although there are recent efforts to spread the supply chain for advanced ICs more broadly which will happen to make it more resilient to attacks.
Basically I mentally see a tree of timelines that all converge on 2 ultimate outcomes, human extinction or humans built ASI. Do you disagree and why?
Humans building AGI ASI likely leads to human extinction.
I disagree: we have many other routes of expansion, including biological improvement, cyborgism, etc. This seems akin to a cultic thinking and akin to Spartan ideas of “only hoplite warfare must be adopted or defeat ensues.”
The “limitations of physics” is quite extensive, and applies even to the pipeline leading up to anything like ASI. I am quite confident that any genuine dedication to the grim game would be more than enough to prevent it, and defiance of it leads to much more likelihood of nuclear winter worlds than ASI dominance.
But I also disagree on your prior of “this world in months”, I suppose we will see in December.
I stated “years if not months”. I agree there is probably not yet enough compute even built to find a true ASI. I assume we will need to explore many cognitive architectures, which means repeating gpt-4 scale training runs thousands of times in order to learn what actually works.
“Months” would be if I am wrong and it’s just a bit of RL away
I find it happy that we probably don’t have enough compute and it is likely this will be restricted even at this fairly early level, long before more extreme measures are needed.
Additionally, I think one should support the Grim Trigger even if you want ASI, because it forces development along more “safe” lines to prevent being Grimmed. It also encourages non-ASI advancement as alternate routes, effectively being a form of regulation.
We will see. There is incredible economic pressure right now to build as much compute as physically possible. Without coordinated government action across all countries capable of building the hardware, this is the default outcome.
We are very close to that world, the time is measured in years if not months.
One bit of timeline arguing: I think odds aren’t zero that we might be on a path that leads to AGI fairly quickly but then ends there and never pushes forward to ASI, not because ASI would be impossible in general, but because we couldn’t reach it this specific way. Our current paradigm isn’t to understand how intelligence works and build it intentionally, it’s to show a big dumb optimizer human solved tasks and tell it “see? We want you to do that”. There’s decent odds that this caps at human potential simply because it can imitate but not surpass its training data, which would require a completely different approach.
Now that I think about it, I think this is basically the path that LLMs likely take, albeit I’d say it caps out a little lower than humans in general. And I give it over 50% probability.
The basic issue here is that the reasoning Transformers do is too inefficient for multi-step problems, and I expect a lot of real world applications of AI outperforming humans will require good multi-step reasoning.
The unexpected success of LLMs isn’t as much about AI progress, as it is about how much our reasoning often is pretty bad in scenarios outside of our ancestral environment. It is less a story of AI progress and more a story of how humans inflate their own strengths like intelligence.
A. It is possible to construct a benchmark to measure if a machine is a general ASI. This would be a very large number of tasks, many simulated though some may be robotic tasks in isolated labs. A general ASI benchmark would have to include tasks humans do not know how to do, but we know how to measure success.
B. We have enough computational resources to train from scratch many ASI level systems so that thousands of attempts are possible. Most attempts would reuse pretrained components in a different architecture.
C. We recursively task the best performing AGIs, as measured by the above benchmark or one meant for weaker systems, to design architectures to perform well on (A)
Currently the best we can do is use RL to design better neural networks, by finding better network architectures and activation functions. Swish was found this way, not sure how much transformer network design came from this type of recursion.
Main idea : the AGI systems exploring possible network architectures are cognitively able to take into account all published research and all past experimental runs, and the ones “in charge” are the ones who demonstrated the most measurable merit at designing prior AGI because they produced the highest performing models on the benchmark.
I think if you think about it you’ll realize it compute were limitless, this AGI to ASI transition you mention could happen instantly. A science fiction story would have it happen in hours. In reality, since training a subhuman system is taking 10k GPUs about 10 days to train, and an AGI will take more—Sam Altman has estimated the compute bill will be close to 100 billion—that’s the limiting factor. You might be right and we stay “stuck” at AGI for years until the resources to discover ASI become available.
I mean, this sounds like a brute force attack to the problem, something that ought not to be very efficient. If our AGI is roughly as smart as the 75th percentile of human engineers it might still just hit its head against a sufficiently hard problem, even in parallel, and especially if we give it the wrong prompt by assuming that the solution will be the extension of current approaches rather than a new one that requires to go back before you can go forward.
You’re correct. In the narrow domain of designing AI architectures you need the system to be at least 1.01 times as good as a human. You want more gain than that because there is a cost to running the system.
Getting gain seems to be trivially easy at least for the types of AI design tasks this has been tried on. Humans are bad at designing network architectures and activation functions.
I theorize that a machine could study the data flows from snapshots from an AI architecture attempting tasks on the AGI/ASI gym, and use that information as well as all previous results to design better architectures.
The last bit is where I expect enormous gain, because the training data set will exceed the amount of data humans can take in in a lifetime, and you would obviously have many smaller “training exercises” to design small systems to build up a general ability. (Enormous early gain. Eventually architectures are going to approach the limits allowed by the underlying compute and datasets)
I think you should reexamine what I said by convergence. Do you...really...think a world that knows how to build (safe, usable tool) ASI would ever be stable by not building it. We are very close to that world, the time is measured in years if not months. Note that any party that gets it working long enough escapes the grim game, they can do whatever they want limited by physics.
I acknowledge your point about chip production, although there are recent efforts to spread the supply chain for advanced ICs more broadly which will happen to make it more resilient to attacks.
Basically I mentally see a tree of timelines that all converge on 2 ultimate outcomes, human extinction or humans built ASI. Do you disagree and why?
Humans building AGI ASI likely leads to human extinction.
I disagree: we have many other routes of expansion, including biological improvement, cyborgism, etc. This seems akin to a cultic thinking and akin to Spartan ideas of “only hoplite warfare must be adopted or defeat ensues.”
The “limitations of physics” is quite extensive, and applies even to the pipeline leading up to anything like ASI. I am quite confident that any genuine dedication to the grim game would be more than enough to prevent it, and defiance of it leads to much more likelihood of nuclear winter worlds than ASI dominance.
But I also disagree on your prior of “this world in months”, I suppose we will see in December.
I stated “years if not months”. I agree there is probably not yet enough compute even built to find a true ASI. I assume we will need to explore many cognitive architectures, which means repeating gpt-4 scale training runs thousands of times in order to learn what actually works.
“Months” would be if I am wrong and it’s just a bit of RL away
I find it happy that we probably don’t have enough compute and it is likely this will be restricted even at this fairly early level, long before more extreme measures are needed.
Additionally, I think one should support the Grim Trigger even if you want ASI, because it forces development along more “safe” lines to prevent being Grimmed. It also encourages non-ASI advancement as alternate routes, effectively being a form of regulation.
We will see. There is incredible economic pressure right now to build as much compute as physically possible. Without coordinated government action across all countries capable of building the hardware, this is the default outcome.
One bit of timeline arguing: I think odds aren’t zero that we might be on a path that leads to AGI fairly quickly but then ends there and never pushes forward to ASI, not because ASI would be impossible in general, but because we couldn’t reach it this specific way. Our current paradigm isn’t to understand how intelligence works and build it intentionally, it’s to show a big dumb optimizer human solved tasks and tell it “see? We want you to do that”. There’s decent odds that this caps at human potential simply because it can imitate but not surpass its training data, which would require a completely different approach.
Now that I think about it, I think this is basically the path that LLMs likely take, albeit I’d say it caps out a little lower than humans in general. And I give it over 50% probability.
The basic issue here is that the reasoning Transformers do is too inefficient for multi-step problems, and I expect a lot of real world applications of AI outperforming humans will require good multi-step reasoning.
The unexpected success of LLMs isn’t as much about AI progress, as it is about how much our reasoning often is pretty bad in scenarios outside of our ancestral environment. It is less a story of AI progress and more a story of how humans inflate their own strengths like intelligence.
Assumptions:
A. It is possible to construct a benchmark to measure if a machine is a general ASI. This would be a very large number of tasks, many simulated though some may be robotic tasks in isolated labs. A general ASI benchmark would have to include tasks humans do not know how to do, but we know how to measure success.
B. We have enough computational resources to train from scratch many ASI level systems so that thousands of attempts are possible. Most attempts would reuse pretrained components in a different architecture.
C. We recursively task the best performing AGIs, as measured by the above benchmark or one meant for weaker systems, to design architectures to perform well on (A)
Currently the best we can do is use RL to design better neural networks, by finding better network architectures and activation functions. Swish was found this way, not sure how much transformer network design came from this type of recursion.
Main idea : the AGI systems exploring possible network architectures are cognitively able to take into account all published research and all past experimental runs, and the ones “in charge” are the ones who demonstrated the most measurable merit at designing prior AGI because they produced the highest performing models on the benchmark.
I think if you think about it you’ll realize it compute were limitless, this AGI to ASI transition you mention could happen instantly. A science fiction story would have it happen in hours. In reality, since training a subhuman system is taking 10k GPUs about 10 days to train, and an AGI will take more—Sam Altman has estimated the compute bill will be close to 100 billion—that’s the limiting factor. You might be right and we stay “stuck” at AGI for years until the resources to discover ASI become available.
I mean, this sounds like a brute force attack to the problem, something that ought not to be very efficient. If our AGI is roughly as smart as the 75th percentile of human engineers it might still just hit its head against a sufficiently hard problem, even in parallel, and especially if we give it the wrong prompt by assuming that the solution will be the extension of current approaches rather than a new one that requires to go back before you can go forward.
You’re correct. In the narrow domain of designing AI architectures you need the system to be at least 1.01 times as good as a human. You want more gain than that because there is a cost to running the system.
Getting gain seems to be trivially easy at least for the types of AI design tasks this has been tried on. Humans are bad at designing network architectures and activation functions.
I theorize that a machine could study the data flows from snapshots from an AI architecture attempting tasks on the AGI/ASI gym, and use that information as well as all previous results to design better architectures.
The last bit is where I expect enormous gain, because the training data set will exceed the amount of data humans can take in in a lifetime, and you would obviously have many smaller “training exercises” to design small systems to build up a general ability. (Enormous early gain. Eventually architectures are going to approach the limits allowed by the underlying compute and datasets)