I don’t think this is a fair reading of Yudkowsky. He was dismissing people who were impressed by the analogy between ANNs and the brain. I’m pretty sure it wasn’t supposed to be a positive claim that ANNs wouldn’t work. Rather, it’s that one couldn’t justifiably believe that they’d work just from the brain analogy, and that if they did work, that would be bad news for what he then called Friendliness (because he was hoping to discover and wield a “clean” theory of intelligence, as contrasted to evolution or gradient descent happening to get there at sufficient scale).
Consider “Artificial Mysterious Intelligence” (2008). In response to someone who said “But neural networks are so wonderful! They solve problems and we don’t have any idea how they do it!”, it’s significant that Yudkowsky’s reply wasn’t, “No, they don’t” (contesting the capabilities claim), but rather, “If you don’t know how your AI works, that is not good. It is bad” (asserting that opaque capabilities are bad for alignment).
One of Yudkowsky’s claims in the post you link is:
It’s hard to build a flying machine if the only thing you understand about flight is that somehow birds magically fly. What you need is a concept of aerodynamic lift, so that you can see how something can fly even if it isn’t exactly like a bird.
This is a claim that lack of the correct mechanistic theory is a formidable barrier for capabilities, not just alignment, and it inaccurately underestimates the amount of empirical understandings available on which to base an empirical approach.
It’s true that it’s hard, even perhaps impossible, to build a flying machine if the only thing you understand is that birds “magically” fly.
But if you are like most people for thousands of years, you’ve observed many types of things flying, gliding, or floating in the air: birds and insects, fabric and leaves, arrows and spears, clouds and smoke.
So if you, like the Montgolfier brothers, observe fabric floating over a fire, and live in an era in which invention is celebrated and have the ability to build, test, and iterate, then you can probably figure out how to build a flying machine without basing this on a fully worked out concept of aerodynamics. Indeed, the Montgolfier brothers thought it was the smoke, rather than the heat, that made their balloons fly. Having the wrong theory was bad, but it didn’t prevent them from building a working hot air balloon.
Let’s try turning Yudkowsky’s quote around:
It’s hard get a concept of aerodynamic lift if the only thing you observe about flight is that somehow birds magically fly. What you need is a rich set of empirical observations and flying mechanisms, so that you can find the common principles for how something can fly even if it isn’t exactly like a bird.
Eliezer went on to list five methods for producing AI that he considered dubious, including builting powerful computers running the most advanced available neural network algorithms, intelligence “emerging from the internet”, and putting “a sufficiently huge quantity of knowledge into [a computer].” But he only admitted that two other methods would work—builting a mechanical duplicate of the human brain and evolving AI via natural selection.
If Eliezer wasn’t meaning to make a confident claim that scaling up neural networks without a fundamental theoretical understanding of intelligence would fail, then he did a poor job of communicating that in these posts. I don’t find that blameworthy—I just think Eliezer comes across as confidently wrong about which avenues would lead to intelligence in these posts, simple as that. He was saying that to achieve a high level of AI capabilities, we’d need a deep mechanistic understanding of how intelligence works akin to our modern understanding of chemistry or aerodynamics, and that didn’t turn out to be the case.
One possible defense is that Eliezer was attacking a weakman, specifically the idea that with only one empirical observation and zero insight into the factors that cause the property of interest (i.e. only seeing that “birds magically fly”), then it’s nearly impossible to replicate that property in a new way. But that’s an uninteresting claim and Eliezer is never uninteresting.
He was saying that to achieve a high level of AI capabilities, we’d need a deep mechanistic understanding of how intelligence works akin to our modern understanding of chemistry or aerodynamics, and that didn’t turn out to be the case.
Another possibility is that at least some people do have a deep mechanistic understanding of how intelligence works, and that’s why they are able to build deep learning systems that ultimately work. Some of the theories of how DL works might be true, and they might be more sophisticated than we are giving credit.
this point continues to be severely underestimated on lesswrong, I think. I had hoped the success of NNs would change this, but it seems people have gone from “we don’t know how NNs work, so they can’t work” to “we don’t know how NNs work, so we can’t trust them”. perhaps we don’t know how they work well enough! there’s lots of mechanistic interpretability work left to do. but we know quite a lot about how they do work and how that relates to human learning.
edit: hmm, people upvoted, then one person with high karma strong downvoted. I’d love to hear that person’s rebuttal, rather than just a strong downvote.
But he only admitted that two other methods would work—builting a mechanical duplicate of the human brain and evolving AI via natural selection.
To be fair, he said that those two will work, and (perhaps?) admitted the possibility of “run advanced neural network algorithms” eventually working. Emphasis mine:
What do all these proposals have in common?
They are all ways to make yourself believe that you can build an Artificial Intelligence, even if you don’t understand exactly how intelligence works.
Agreed. The right interpretation there is methods 4 and 5 are ~guaranteed to work, given sufficient resources and time, while methods 1-3 less than guaranteed to work. I stand by my claim that EY was clearly projecting confident doubt that neural networks would achieve intelligence without a deep theoretical understanding of intelligence in these posts. I think I underemphasized the implication of this passage that methods 1-3 could possibly work, but I think I accurately assessed the tone of extreme skepticism on EY’s part.
With the enormous benefit of 15 years of hindsight, we can now say that message was misleading or mistaken, take your pick. As I say, I wouldn’t find fault with Eliezer or anyone who believed him at the time for making this mistake; I didn’t even have an opinion at the time, much less an interesting mistake! I would only find fault with attempts to stretch the argument and portray him as “technically not wrong” in some uninteresting sense.
I think it might be relevant to note here that it’s not really humans who are building current SOTA AIs—rather, it’s some optimizer like SGD that’s doing most of the work. SGD does not have any mechanistic understanding of intelligence (nor anything else). And indeed, it takes a heck of a lot of data and compute for SGD to build those AIs. This seems to be in line with Yudkowsky’s claim that it’s hard/inefficient to build something without understanding it.
If Eliezer wasn’t meaning to make a confident claim that scaling up neural networks without a fundamental theoretical understanding of intelligence would fail, then [...]
I think it’s important to distinguish between
Scaling up a neural network, and running some kind of fixed algorithm on it.
Scaling up a neural network, and using SGD to optimize the parameters of the NN, so that the NN ends up learning a whole new set of algorithms.
IIUC, in Artificial Mysterious Intelligence, Yudkowsky seemed to be saying that the former would probably fail. OTOH, I don’t know what kinds of NN algorithms were popular back in 2008, or exactly what NN algorithms Yudkowsky was referring to, so… *shrugs*.
If that were the case, I actually would fault Eliezer, at least a little. He’s frequently, though by no means always, stuck to qualitative and hard-to-pin-down punditry like we see here, rather than to unambiguous forecasting.
This allows him, or his defenders, to retroactively defend his predictions as somehow correct even when they seem wrong in hindsight.
Let’s imagine for a moment that Eliezer’s right that AI safety is a cosmically important issue, and yet that he’s quite mistaken about all the technical details of how AGI will arise and how to effectively make it safe. It would be important to know whether we can trust his judgment and leadership.
Without the ability to evaluate his performance, either by going with the most obvious interpretation of his qualitative judgments or an unambiguous forecast, it’s hard to evaluate his performance as an AI safety leader. Combine that with a culture of deference to perceived expertise and status and the problem gets worse.
So I prioritize the avoidance of special pleading in this case: I think Eliezer comes across as clearly wrong in substance in this specific post, and that it’s important not to reach for ways “he was actually right from a certain point of view” when evaluating his predictive accuracy.
Similarly, I wouldn’t judge as correct the early COVID-19 pronouncements that masks don’t work to stop the spread just because cloth masks are poor-to-ineffective and many people refuse to wear masks properly. There’s a way we can stretch the interpretation to make them seem sort of right, but we shouldn’t. We should expect public health messaging to be clearly right in substance, if it’s not making cut and dry unambiguous quantitative forecasts but is instead delivering qualitative judgments of efficacy.
None of that bears on how easy or hard it was to build gpt-4. It only bears on how we should evaluate Eliezer as a forecaster/pundit/AI safety leader.
I think several things here, considering the broader thread:
You’ve done a great job in communicating several reactions I also had:
There are signs of serious mispredictions and mistakes in some of the 2008 posts.
There are ways to read these posts as not that bad in hindsight, but we should be careful in giving too much benefit of the doubt.
Overall these observations constitute important evidence on EY’s alignment intuitions and ability to make qualitative AI predictions.
I did a bad job of marking my interpretations of what Eliezer wrote, as opposed to claiming he did dismiss ANNs. Hopefully my edits have fixed my mistakes.
I don’t think this is a fair reading of Yudkowsky. He was dismissing people who were impressed by the analogy between ANNs and the brain. I’m pretty sure it wasn’t supposed to be a positive claim that ANNs wouldn’t work. Rather, it’s that one couldn’t justifiably believe that they’d work just from the brain analogy, and that if they did work, that would be bad news for what he then called Friendliness (because he was hoping to discover and wield a “clean” theory of intelligence, as contrasted to evolution or gradient descent happening to get there at sufficient scale).
Consider “Artificial Mysterious Intelligence” (2008). In response to someone who said “But neural networks are so wonderful! They solve problems and we don’t have any idea how they do it!”, it’s significant that Yudkowsky’s reply wasn’t, “No, they don’t” (contesting the capabilities claim), but rather, “If you don’t know how your AI works, that is not good. It is bad” (asserting that opaque capabilities are bad for alignment).
One of Yudkowsky’s claims in the post you link is:
This is a claim that lack of the correct mechanistic theory is a formidable barrier for capabilities, not just alignment, and it inaccurately underestimates the amount of empirical understandings available on which to base an empirical approach.
It’s true that it’s hard, even perhaps impossible, to build a flying machine if the only thing you understand is that birds “magically” fly.
But if you are like most people for thousands of years, you’ve observed many types of things flying, gliding, or floating in the air: birds and insects, fabric and leaves, arrows and spears, clouds and smoke.
So if you, like the Montgolfier brothers, observe fabric floating over a fire, and live in an era in which invention is celebrated and have the ability to build, test, and iterate, then you can probably figure out how to build a flying machine without basing this on a fully worked out concept of aerodynamics. Indeed, the Montgolfier brothers thought it was the smoke, rather than the heat, that made their balloons fly. Having the wrong theory was bad, but it didn’t prevent them from building a working hot air balloon.
Let’s try turning Yudkowsky’s quote around:
Eliezer went on to list five methods for producing AI that he considered dubious, including builting powerful computers running the most advanced available neural network algorithms, intelligence “emerging from the internet”, and putting “a sufficiently huge quantity of knowledge into [a computer].” But he only admitted that two other methods would work—builting a mechanical duplicate of the human brain and evolving AI via natural selection.
If Eliezer wasn’t meaning to make a confident claim that scaling up neural networks without a fundamental theoretical understanding of intelligence would fail, then he did a poor job of communicating that in these posts. I don’t find that blameworthy—I just think Eliezer comes across as confidently wrong about which avenues would lead to intelligence in these posts, simple as that. He was saying that to achieve a high level of AI capabilities, we’d need a deep mechanistic understanding of how intelligence works akin to our modern understanding of chemistry or aerodynamics, and that didn’t turn out to be the case.
One possible defense is that Eliezer was attacking a weakman, specifically the idea that with only one empirical observation and zero insight into the factors that cause the property of interest (i.e. only seeing that “birds magically fly”), then it’s nearly impossible to replicate that property in a new way. But that’s an uninteresting claim and Eliezer is never uninteresting.
Another possibility is that at least some people do have a deep mechanistic understanding of how intelligence works, and that’s why they are able to build deep learning systems that ultimately work. Some of the theories of how DL works might be true, and they might be more sophisticated than we are giving credit.
this point continues to be severely underestimated on lesswrong, I think. I had hoped the success of NNs would change this, but it seems people have gone from “we don’t know how NNs work, so they can’t work” to “we don’t know how NNs work, so we can’t trust them”. perhaps we don’t know how they work well enough! there’s lots of mechanistic interpretability work left to do. but we know quite a lot about how they do work and how that relates to human learning.
edit: hmm, people upvoted, then one person with high karma strong downvoted. I’d love to hear that person’s rebuttal, rather than just a strong downvote.
To be fair, he said that those two will work, and (perhaps?) admitted the possibility of “run advanced neural network algorithms” eventually working. Emphasis mine:
Agreed. The right interpretation there is methods 4 and 5 are ~guaranteed to work, given sufficient resources and time, while methods 1-3 less than guaranteed to work. I stand by my claim that EY was clearly projecting confident doubt that neural networks would achieve intelligence without a deep theoretical understanding of intelligence in these posts. I think I underemphasized the implication of this passage that methods 1-3 could possibly work, but I think I accurately assessed the tone of extreme skepticism on EY’s part.
With the enormous benefit of 15 years of hindsight, we can now say that message was misleading or mistaken, take your pick. As I say, I wouldn’t find fault with Eliezer or anyone who believed him at the time for making this mistake; I didn’t even have an opinion at the time, much less an interesting mistake! I would only find fault with attempts to stretch the argument and portray him as “technically not wrong” in some uninteresting sense.
Ok, I guess I just read Eliezer as saying something uninteresting with a touch of negative sentiment towards neural nets.
I think it might be relevant to note here that it’s not really humans who are building current SOTA AIs—rather, it’s some optimizer like SGD that’s doing most of the work. SGD does not have any mechanistic understanding of intelligence (nor anything else). And indeed, it takes a heck of a lot of data and compute for SGD to build those AIs. This seems to be in line with Yudkowsky’s claim that it’s hard/inefficient to build something without understanding it.
I think it’s important to distinguish between
Scaling up a neural network, and running some kind of fixed algorithm on it.
Scaling up a neural network, and using SGD to optimize the parameters of the NN, so that the NN ends up learning a whole new set of algorithms.
IIUC, in Artificial Mysterious Intelligence, Yudkowsky seemed to be saying that the former would probably fail. OTOH, I don’t know what kinds of NN algorithms were popular back in 2008, or exactly what NN algorithms Yudkowsky was referring to, so… *shrugs*.
If that were the case, I actually would fault Eliezer, at least a little. He’s frequently, though by no means always, stuck to qualitative and hard-to-pin-down punditry like we see here, rather than to unambiguous forecasting.
This allows him, or his defenders, to retroactively defend his predictions as somehow correct even when they seem wrong in hindsight.
Let’s imagine for a moment that Eliezer’s right that AI safety is a cosmically important issue, and yet that he’s quite mistaken about all the technical details of how AGI will arise and how to effectively make it safe. It would be important to know whether we can trust his judgment and leadership.
Without the ability to evaluate his performance, either by going with the most obvious interpretation of his qualitative judgments or an unambiguous forecast, it’s hard to evaluate his performance as an AI safety leader. Combine that with a culture of deference to perceived expertise and status and the problem gets worse.
So I prioritize the avoidance of special pleading in this case: I think Eliezer comes across as clearly wrong in substance in this specific post, and that it’s important not to reach for ways “he was actually right from a certain point of view” when evaluating his predictive accuracy.
Similarly, I wouldn’t judge as correct the early COVID-19 pronouncements that masks don’t work to stop the spread just because cloth masks are poor-to-ineffective and many people refuse to wear masks properly. There’s a way we can stretch the interpretation to make them seem sort of right, but we shouldn’t. We should expect public health messaging to be clearly right in substance, if it’s not making cut and dry unambiguous quantitative forecasts but is instead delivering qualitative judgments of efficacy.
None of that bears on how easy or hard it was to build gpt-4. It only bears on how we should evaluate Eliezer as a forecaster/pundit/AI safety leader.
I think several things here, considering the broader thread:
You’ve done a great job in communicating several reactions I also had:
There are signs of serious mispredictions and mistakes in some of the 2008 posts.
There are ways to read these posts as not that bad in hindsight, but we should be careful in giving too much benefit of the doubt.
Overall these observations constitute important evidence on EY’s alignment intuitions and ability to make qualitative AI predictions.
I did a bad job of marking my interpretations of what Eliezer wrote, as opposed to claiming he did dismiss ANNs. Hopefully my edits have fixed my mistakes.