Solving intelligence is the most important problem we face because intelligence can be used to solve everything else
We know it’s possible to solve intelligence because evolution has already done it
We’ve made enormous progress towards solving intelligence in the last few years
Given our situation, it’s surprising that the broader world doesn’t share LessWrong’s fascination with AGI. If people were giving it the weight it deserved, it would overshadow even the global culture war which has dominated the airwaves for the last 5 years and pulled in so many of us.
And forget, for a moment, about true intelligence. The narrow AIs that we already have are going to shake up our lives in a big way in the immediate future. The days of the art industry have been numbered ever since the announcement of DALLE-1.
Unfortunately, despite my general enthusiasm for machine learning, I have to take issue with the common sentiment on here that AGI is right around the corner. I think it’s very unlikely (<10% chance) that we’ll see AGI within the next 50 years, and entirely possible (>25% chance) that it will take over 500 years.
For any human who can do any job, there is a computer program (not necessarily the same one every time) that can do the same job for $25/hr or less.
Wishful thinking
My core thesis is that the people on this site are collectively falling victim to wishful thinking. Most of them really want to experience transformative AGI for themselves, and this is causing them to make unreasonably aggressive predictions.
The first clue that wishful thinking might be at play is that everyone’s forecasts seem to put AGI smack in the middle of their lifetimes. It’s very common to see predictions of AGI in 20 years, but I don’t think I’ve ever seen a prediction in the 200-500 year range. This is strange, because in the broader context of human progress, AGI in 2040 and AGI in 2240 are not significantly different. Those years are side-by-side in a zoomed-out view of human history.
I’m reminded of Ray Kurzweil and Aubrey de Grey, who conveniently predict that immortality will be achieved right around the time they reach old age. Both of those men have a long enough history of predictions that we can now say pretty confidently they’ve been overoptimistic.
More generally, it’s hard to find anyone who both:
Feels that aging is bad; and
Considers it extremely unlikely that aging will be solved in their lifetime
even though both of those seem like the obvious, default positions to take.
When we desperately want to see a transformative technology, be it immortality or AGI, there is a strong pressure on us to believe that we will. But the universe doesn’t care about what we want. You could die 64 and a half years before the creation of AGI, and no laws of physics (or laws of scaling) would be violated.
But isn’t LessWrong afraid of AGI?
But hang on, the people on LessWrong don’t want to experience AGI because they believe it has a good chance of destroying the world. So how could AGI in the near future possibly constitute a wishful thinking scenario?
If you don’t think you would like to witness AGI in your lifetime, you may want to look closer at your feelings. Imagine you’re talking to a sage who knows humanity’s future. You ask him about the arrival of AGI. Which of these responses would you prefer?
AGI doesn’t appear until a few hundred years after your death.
AGI appears in 2040, and the sage can’t see what happens beyond that point.
For me, the first creates a feeling of dread, while the second creates a feeling of excitement. Please genuinely introspect and decide which revelation you would prefer. If it is indeed the second, and you also believe there is a decent chance of AGI in your lifetime, then you must be very careful that you are not falling prey to wishful thinking.
As a specific example of what I suspect is a bit of cognitive dissonance, look at the recent post on AGI by porby, which predicts AGI by 2030. I loved reading that post because it promises that the future is going to be wild. If porby is right, we’re all in for an adventure. Based on the breathless tone of the post, I would surmise that porby is as excited by his conclusion as I am. For example, we have this excerpt:
This is crazy! I’m raising my eyebrows right now to emphasize it! Consider also doing so! This is weird enough to warrant it!
Would you have predicted this in 2016? I don’t think I would have!
Does this strike you as someone who dreads the arrival of AGI? It seems to me like he is awaiting it with great anticipation.
But then in the comments on the post, he says that he hopes he’s wrong about AGI! If you’re reading this porby, do you really want to be wrong?
One of the reasons I come to this site is because, when I read between the lines, it reassures me that we are headed for an idyllic—or at least very interesting—future. I don’t think that makes me a rare exception. I think it’s more likely that there is an echo chamber in effect, where everyone reinforces each other’s hopes for a transhuman future. And if that is indeed the case, then this is an environment ripe for wishful thinking about the pace of technological progress.
The object-level question
The wishful thinking argument may be dissatisfying because it doesn’t explain why AGI is likely to take more than 50 years. It merely explains why such a large number of people in these circles are saying otherwise. I will now directly explain why I do not expect AGI to arrive in the next several decades.
There are two paths by which we can reach AGI. We can study and replicate the core features of the brain (the neuroscience approach) or we can keep incrementally improving our machine learning models without explicitly copying the brain (the machine learning approach).
The neuroscience approach
The neuroscience approach is, as far as I can tell, extremely unlikely to bear fruit within the next 50 years.
My impression of neuroscientists is that they understand the brain decently well at a low level. Here’s how the neurons are wired together; here are the conditions under which they fire; here are the groups of neurons that tend to fire when a person is afraid or dreaming or speaking. But they can’t say anything coherent about its high-level operation, which is where the secrets of intelligence are really contained.
I will also note the air of mystery that hangs thick around psychoactive drugs and psychiatric disorders. How does general anesthesia induce unconsciousness? Why do anti-depressants work? What specifically is going wrong in a brain with anxiety or BPD or schizophrenia? Ask any of these questions and you will get a mumbled answer about The gaba-D3-transmitters and the dopa-B4-receptors.
It’s like they’re trying to figure out how a computer creates a web browser, and all they can tell you is that when you open a new tab, a lot of electrical signals begin to pass through the wires in the F-53 region of the motherboard. This is not what a mature scientific field looks like.
And even what little *is* known is constantly changing. It seems like every month I see a new article about how some discovery has rocked the field of neuroscience, and actually the brain is doing something far more sophisticated than they thought.
Biology is hard. Our understanding of the rest of the human body isn’t much better. Medicine is still mostly guesswork. If you go to three doctors, you’ll get three different diagnoses. We can’t cause a body to lose fat or grow hair despite massive financial incentives to do so.
We don’t even understand C. Elegans well enough to simulate it, and that is a microscopic worm with 302 neurons and less than a thousand cells.
Does this seem to you like a civilization that is on the verge of solving the greatest problem in biology?
The machine learning approach
Neuroscientists are not likely to hand us the key to AGI any time soon, so if we want it by 2072, we’re going to have to find it ourselves. This means we try things out, think really hard, adjust our approach, and repeat. Eventually we find the correct series of novel insights that lead to AGI (John Carmack predicts we will need 6).
Notice that this a math problem, not an engineering problem. The internet was an engineering problem. The technology was essentially there and it was just a matter of putting it together in the right way. The Manhattan Project and the metaverse are engineering problems. Solving intelligence is more like proving the Riemann Hypothesis, where we don’t have any clue how much work it’s going to take.
This is what the arguments in favor of imminent AGI ignore. They just look at the graph of our available computing power, find where it crosses the power of the human brain, and assume we will get AGI around that date. They’re sweeping all of the math work—all of the necessary algorithmic innovations—under the rug. As if that stuff will just fall into our lap, ready to copy into PyTorch.
But creative insights do not come on command. It’s not unheard of that a math problem remains open for 1000 years. Geoffrey Hinton—and a cohort of other researchers—has spent the last 50 years trying to figure out intelligence with only partial success. Physicists have been seeking a theory of everything for hundreds of years and have not yet found one.
Endeavors like these require us to explore many branches before we find the right one. We can easily lose 30 years to a seductive dead end. Or a field can fall into a multi-decade winter until it is revived by a maverick who overturns the prevailing orthodoxy. 50 years is the blink of an eye as far as these grand challenges of understanding go. It seems bizarrely overconfident to expect total victory in such a short timeframe.
And that’s just the mathematical/theoretical part of the problem. Once we’ve guessed an algorithm that will in fact produce an AGI, it may still be prohibitively hard to run. We may not have enough data. Or it may need to be trained in a very complicated simulator, or worse, in the real world where materials are expensive and the speed of time is fixed.
Current state of the art
Some people say that we’ve already had the vast majority of the creative insights that are needed for AGI. For example, they argue that GPT-3 can be made into AGI with a little bit of tweaking and scaling.
Rub the stars out of your eyes for a second. GPT-3 is a huge leap forward, but it still has some massive structural deficiencies. From most to least important:
It doesn’t care whether it says correct things, only whether it completes its prompts in a realistic way
It can’t choose to spend extra computation on more difficult prompts
It has no memory outside of its current prompt
It can’t take advantage of external resources (like using a text file to organize its thoughts, or using a calculator for arithmetic)
It can’t think unless it’s processing a prompt
It doesn’t know that it’s a machine learning model
“But these can be solved with a layer of prompt engineering!” Give me a break. That’s obviously a brittle solution that does not address the underlying issues.
You can give your pet theories as to how these limitations can be repaired, but it’s not worth much until someone actually writes the code. Before then, we can’t know how difficult it will be or how many years it will take. It’s possible that the whole paradigm behind GPT-3 is flawed in some basic way that prevents us from solving these problems, and we will only reach AGI when we go back and rethink the foundations.
And maybe we will create a program with none of these flaws that is still lacking some necessary aspect of intelligence that I’ve failed to list. There are just too many unknowns here to be at all confident of AGI in the near future.
There is perhaps a better case to be made that MuZero is nearing AGI. It has a different and arguably less serious set of limitations than GPT-3. But it has only been applied to tasks like Go and Atari. Compared to the task of, say, working as a software engineer, these are miniscule toy problems.
With both GPT-3 and MuZero, there is a clearly a chasm yet to be crossed, and that chasm can hide any number of multi-decade subchallenges. Looking at the eye-watering distance between state of the art AI and human intelligence, I think it’s unreasonable to assume we’ll cross that chasm in the next 50 years.
Conclusion
I regret having to take the side of pessimism for this article. It’s really not my natural disposition when it comes to technology. Even though I don’t anticipate living to see AGI, I will reiterate that we are in for quite a ride with narrow AI. We’ve already seen some true magic since the dawn of the deep learning revolution, and it only gets weirder from here. It’s such a privilege to be able to witness it.
Sorry if any of this came off as a bit rude. I know some people would prefer if I had skipped the psychoanalytic angle of wishful thinking, and just directly made my case for a longer AGI timeline. However, I think it’s important, because when we look back in 2070 and AGI remains elusive, that will be the clearest explanation of what went wrong in the heady days of 2022.
AGI in our lifetimes is wishful thinking
A few facts:
Solving intelligence is the most important problem we face because intelligence can be used to solve everything else
We know it’s possible to solve intelligence because evolution has already done it
We’ve made enormous progress towards solving intelligence in the last few years
Given our situation, it’s surprising that the broader world doesn’t share LessWrong’s fascination with AGI. If people were giving it the weight it deserved, it would overshadow even the global culture war which has dominated the airwaves for the last 5 years and pulled in so many of us.
And forget, for a moment, about true intelligence. The narrow AIs that we already have are going to shake up our lives in a big way in the immediate future. The days of the art industry have been numbered ever since the announcement of DALLE-1.
Unfortunately, despite my general enthusiasm for machine learning, I have to take issue with the common sentiment on here that AGI is right around the corner. I think it’s very unlikely (<10% chance) that we’ll see AGI within the next 50 years, and entirely possible (>25% chance) that it will take over 500 years.
To be precise, let’s use the Future Fund’s definition of AGI:
Wishful thinking
My core thesis is that the people on this site are collectively falling victim to wishful thinking. Most of them really want to experience transformative AGI for themselves, and this is causing them to make unreasonably aggressive predictions.
The first clue that wishful thinking might be at play is that everyone’s forecasts seem to put AGI smack in the middle of their lifetimes. It’s very common to see predictions of AGI in 20 years, but I don’t think I’ve ever seen a prediction in the 200-500 year range. This is strange, because in the broader context of human progress, AGI in 2040 and AGI in 2240 are not significantly different. Those years are side-by-side in a zoomed-out view of human history.
I’m reminded of Ray Kurzweil and Aubrey de Grey, who conveniently predict that immortality will be achieved right around the time they reach old age. Both of those men have a long enough history of predictions that we can now say pretty confidently they’ve been overoptimistic.
More generally, it’s hard to find anyone who both:
Feels that aging is bad; and
Considers it extremely unlikely that aging will be solved in their lifetime
even though both of those seem like the obvious, default positions to take.
When we desperately want to see a transformative technology, be it immortality or AGI, there is a strong pressure on us to believe that we will. But the universe doesn’t care about what we want. You could die 64 and a half years before the creation of AGI, and no laws of physics (or laws of scaling) would be violated.
But isn’t LessWrong afraid of AGI?
But hang on, the people on LessWrong don’t want to experience AGI because they believe it has a good chance of destroying the world. So how could AGI in the near future possibly constitute a wishful thinking scenario?
If you don’t think you would like to witness AGI in your lifetime, you may want to look closer at your feelings. Imagine you’re talking to a sage who knows humanity’s future. You ask him about the arrival of AGI. Which of these responses would you prefer?
AGI doesn’t appear until a few hundred years after your death.
AGI appears in 2040, and the sage can’t see what happens beyond that point.
For me, the first creates a feeling of dread, while the second creates a feeling of excitement. Please genuinely introspect and decide which revelation you would prefer. If it is indeed the second, and you also believe there is a decent chance of AGI in your lifetime, then you must be very careful that you are not falling prey to wishful thinking.
As a specific example of what I suspect is a bit of cognitive dissonance, look at the recent post on AGI by porby, which predicts AGI by 2030. I loved reading that post because it promises that the future is going to be wild. If porby is right, we’re all in for an adventure. Based on the breathless tone of the post, I would surmise that porby is as excited by his conclusion as I am. For example, we have this excerpt:
Does this strike you as someone who dreads the arrival of AGI? It seems to me like he is awaiting it with great anticipation.
But then in the comments on the post, he says that he hopes he’s wrong about AGI! If you’re reading this porby, do you really want to be wrong?
One of the reasons I come to this site is because, when I read between the lines, it reassures me that we are headed for an idyllic—or at least very interesting—future. I don’t think that makes me a rare exception. I think it’s more likely that there is an echo chamber in effect, where everyone reinforces each other’s hopes for a transhuman future. And if that is indeed the case, then this is an environment ripe for wishful thinking about the pace of technological progress.
The object-level question
The wishful thinking argument may be dissatisfying because it doesn’t explain why AGI is likely to take more than 50 years. It merely explains why such a large number of people in these circles are saying otherwise. I will now directly explain why I do not expect AGI to arrive in the next several decades.
There are two paths by which we can reach AGI. We can study and replicate the core features of the brain (the neuroscience approach) or we can keep incrementally improving our machine learning models without explicitly copying the brain (the machine learning approach).
The neuroscience approach
The neuroscience approach is, as far as I can tell, extremely unlikely to bear fruit within the next 50 years.
My impression of neuroscientists is that they understand the brain decently well at a low level. Here’s how the neurons are wired together; here are the conditions under which they fire; here are the groups of neurons that tend to fire when a person is afraid or dreaming or speaking. But they can’t say anything coherent about its high-level operation, which is where the secrets of intelligence are really contained.
I will also note the air of mystery that hangs thick around psychoactive drugs and psychiatric disorders. How does general anesthesia induce unconsciousness? Why do anti-depressants work? What specifically is going wrong in a brain with anxiety or BPD or schizophrenia? Ask any of these questions and you will get a mumbled answer about The gaba-D3-transmitters and the dopa-B4-receptors.
It’s like they’re trying to figure out how a computer creates a web browser, and all they can tell you is that when you open a new tab, a lot of electrical signals begin to pass through the wires in the F-53 region of the motherboard. This is not what a mature scientific field looks like.
And even what little *is* known is constantly changing. It seems like every month I see a new article about how some discovery has rocked the field of neuroscience, and actually the brain is doing something far more sophisticated than they thought.
Biology is hard. Our understanding of the rest of the human body isn’t much better. Medicine is still mostly guesswork. If you go to three doctors, you’ll get three different diagnoses. We can’t cause a body to lose fat or grow hair despite massive financial incentives to do so.
We don’t even understand C. Elegans well enough to simulate it, and that is a microscopic worm with 302 neurons and less than a thousand cells.
Does this seem to you like a civilization that is on the verge of solving the greatest problem in biology?
The machine learning approach
Neuroscientists are not likely to hand us the key to AGI any time soon, so if we want it by 2072, we’re going to have to find it ourselves. This means we try things out, think really hard, adjust our approach, and repeat. Eventually we find the correct series of novel insights that lead to AGI (John Carmack predicts we will need 6).
Notice that this a math problem, not an engineering problem. The internet was an engineering problem. The technology was essentially there and it was just a matter of putting it together in the right way. The Manhattan Project and the metaverse are engineering problems. Solving intelligence is more like proving the Riemann Hypothesis, where we don’t have any clue how much work it’s going to take.
This is what the arguments in favor of imminent AGI ignore. They just look at the graph of our available computing power, find where it crosses the power of the human brain, and assume we will get AGI around that date. They’re sweeping all of the math work—all of the necessary algorithmic innovations—under the rug. As if that stuff will just fall into our lap, ready to copy into PyTorch.
But creative insights do not come on command. It’s not unheard of that a math problem remains open for 1000 years. Geoffrey Hinton—and a cohort of other researchers—has spent the last 50 years trying to figure out intelligence with only partial success. Physicists have been seeking a theory of everything for hundreds of years and have not yet found one.
Endeavors like these require us to explore many branches before we find the right one. We can easily lose 30 years to a seductive dead end. Or a field can fall into a multi-decade winter until it is revived by a maverick who overturns the prevailing orthodoxy. 50 years is the blink of an eye as far as these grand challenges of understanding go. It seems bizarrely overconfident to expect total victory in such a short timeframe.
And that’s just the mathematical/theoretical part of the problem. Once we’ve guessed an algorithm that will in fact produce an AGI, it may still be prohibitively hard to run. We may not have enough data. Or it may need to be trained in a very complicated simulator, or worse, in the real world where materials are expensive and the speed of time is fixed.
Current state of the art
Some people say that we’ve already had the vast majority of the creative insights that are needed for AGI. For example, they argue that GPT-3 can be made into AGI with a little bit of tweaking and scaling.
Rub the stars out of your eyes for a second. GPT-3 is a huge leap forward, but it still has some massive structural deficiencies. From most to least important:
It doesn’t care whether it says correct things, only whether it completes its prompts in a realistic way
It can’t choose to spend extra computation on more difficult prompts
It has no memory outside of its current prompt
It can’t take advantage of external resources (like using a text file to organize its thoughts, or using a calculator for arithmetic)
It can’t think unless it’s processing a prompt
It doesn’t know that it’s a machine learning model
“But these can be solved with a layer of prompt engineering!” Give me a break. That’s obviously a brittle solution that does not address the underlying issues.
You can give your pet theories as to how these limitations can be repaired, but it’s not worth much until someone actually writes the code. Before then, we can’t know how difficult it will be or how many years it will take. It’s possible that the whole paradigm behind GPT-3 is flawed in some basic way that prevents us from solving these problems, and we will only reach AGI when we go back and rethink the foundations.
And maybe we will create a program with none of these flaws that is still lacking some necessary aspect of intelligence that I’ve failed to list. There are just too many unknowns here to be at all confident of AGI in the near future.
There is perhaps a better case to be made that MuZero is nearing AGI. It has a different and arguably less serious set of limitations than GPT-3. But it has only been applied to tasks like Go and Atari. Compared to the task of, say, working as a software engineer, these are miniscule toy problems.
With both GPT-3 and MuZero, there is a clearly a chasm yet to be crossed, and that chasm can hide any number of multi-decade subchallenges. Looking at the eye-watering distance between state of the art AI and human intelligence, I think it’s unreasonable to assume we’ll cross that chasm in the next 50 years.
Conclusion
I regret having to take the side of pessimism for this article. It’s really not my natural disposition when it comes to technology. Even though I don’t anticipate living to see AGI, I will reiterate that we are in for quite a ride with narrow AI. We’ve already seen some true magic since the dawn of the deep learning revolution, and it only gets weirder from here. It’s such a privilege to be able to witness it.
Sorry if any of this came off as a bit rude. I know some people would prefer if I had skipped the psychoanalytic angle of wishful thinking, and just directly made my case for a longer AGI timeline. However, I think it’s important, because when we look back in 2070 and AGI remains elusive, that will be the clearest explanation of what went wrong in the heady days of 2022.