In my view, the biological anchors and the Very Serious estimates derived therefrom are really useful for the following very narrow yet plausibly impactful purpose
I don’t understand why it’s not just useful directly. Saying that the numbers are not true upper or lower bounds seems like it’s expecting way too much!
They’re not even labeled as bounds (at least in the headline). They’re supposed to be “anchors”.
Suppose you’d never done the analysis to know how much compute a human brain uses, or how much compute all of evolution had used. Wouldn’t this report be super useful to you?
Sure, it doesn’t directly tell you when TAI is going to come, because there’s a separate thing you don’t know, which is how compute-efficient our systems are going to be compared to the human brain. And also that translation factor is changing with time. But surely that’s another quantity we can have a distribution over.
If there’s some quantity that we don’t know the value of, but we have at least one way to estimate it using some other uncertain quantities, why is it not useful to reduce our uncertainty about some of those other quantities?
This seems like exactly the kind of thing superforecasters are supposed to do. Or that an Eliezer-informed Bayesian rationalist is supposed to do. Quantify your uncertainty. Don’t be afraid to use a probability distribution. Don’t throw away relevant information, but instead use it to reduce your uncertainty and update your probabilities.
If Eliezer’s point is just that the report shouldn’t be taken as the gospel truth of when AI is going to come, then fine. Or if he just wants to highlight that there’s still uncertainty over the translation factor between the brain’s compute-efficiency and our ML systems’ compute-efficiency, then that seems like a good point too.
But I don’t really understand the point of the rest of the article. If I wanted to have any idea at all when TAI might come, then Moravec’s 1988 calculations regarding the human brain seem super interesting. And also Somebody on the Internet’s 2006 calculation of how much compute evolution had used.
Either of them would be wrong to think that their number precisely pins down the date. But if you started out not knowing whether to expect AGI in one year or in 10,000 years, then it seems like learning the human brain number and the all-of-evolution number should radically reduce your uncertainty.
It still doesn’t reduce your uncertainty all the way, because we still don’t know the compute-efficiency translation factor. But who said it reduced uncertainty all the way? Not OpenPhil.
Eliezer’s main point of his ~20k words isn’t really what I want to defend, but I will say a few words about how I would constructively interpret it. I think Eliezer’s main claim is that he has an intuitive system-1 model of the inhomogeneous Poisson process that emits working AGI systems, and that this model isn’t informed by the compute equivalents of biological anchors, and that his lack of being informed by that isn’t a mistake. I’m not sure if he’s actually making the stronger claim that anyone whose model is informed by the biological anchors is making a mistake, but if so, I don’t agree. My own model is somewhat informed by biological anchors; it’s more informed by what the TAI report calls “subjective impressiveness extrapolation”, extrapolations on benchmark performance, and some vague sense of other point processes that emit AI winters and major breakthroughs. Someone who has total Knightean uncertainty and no intuitive models of how AGI comes about would surely do well to adopt OpenPhil’s distribution as a prior.
I suppose this kind of report is less useful to you the more you think the uncertainty lies in the compute-efficiency translation factor variable. If you think most of the orders of magnitude are in that value, you don’t care so much about the biological anchors.
And maybe you’re in that state if you think building AGI is just a matter of coming up with clever algorithms. But if you think there’s just some as-yet undiscovered general reasoning algorithm, and that’s really the only thing that matters for AGI, then why are you at all impressed by increasingly capable AI systems that use more and more compute, like AlphaGo or GPT-3? It’s (supposedly) not general reasoning, so why does it matter?
It seems to me like the compute-efficiency translation factor is just a perfectly reasonable non-mysterious quantity that we can also get information about and estimate. It’s not going to be the same factor across all tasks, but it seems like we could at least get some idea of what quantities it plausibly could take on by looking at how much compute current (and old) systems are using to match human performance for various tasks, and looking at how that number varies across tasks and changes over time.
I wouldn’t expect such analysis to leave the translation factor so uncertain that our total uncertainty is concentrated so overwhelmingly in that parameter that the biological anchors become useless.
I don’t understand why it’s not just useful directly. Saying that the numbers are not true upper or lower bounds seems like it’s expecting way too much!
They’re not even labeled as bounds (at least in the headline). They’re supposed to be “anchors”.
Suppose you’d never done the analysis to know how much compute a human brain uses, or how much compute all of evolution had used. Wouldn’t this report be super useful to you?
Sure, it doesn’t directly tell you when TAI is going to come, because there’s a separate thing you don’t know, which is how compute-efficient our systems are going to be compared to the human brain. And also that translation factor is changing with time. But surely that’s another quantity we can have a distribution over.
If there’s some quantity that we don’t know the value of, but we have at least one way to estimate it using some other uncertain quantities, why is it not useful to reduce our uncertainty about some of those other quantities?
This seems like exactly the kind of thing superforecasters are supposed to do. Or that an Eliezer-informed Bayesian rationalist is supposed to do. Quantify your uncertainty. Don’t be afraid to use a probability distribution. Don’t throw away relevant information, but instead use it to reduce your uncertainty and update your probabilities.
If Eliezer’s point is just that the report shouldn’t be taken as the gospel truth of when AI is going to come, then fine. Or if he just wants to highlight that there’s still uncertainty over the translation factor between the brain’s compute-efficiency and our ML systems’ compute-efficiency, then that seems like a good point too.
But I don’t really understand the point of the rest of the article. If I wanted to have any idea at all when TAI might come, then Moravec’s 1988 calculations regarding the human brain seem super interesting. And also Somebody on the Internet’s 2006 calculation of how much compute evolution had used.
Either of them would be wrong to think that their number precisely pins down the date. But if you started out not knowing whether to expect AGI in one year or in 10,000 years, then it seems like learning the human brain number and the all-of-evolution number should radically reduce your uncertainty.
It still doesn’t reduce your uncertainty all the way, because we still don’t know the compute-efficiency translation factor. But who said it reduced uncertainty all the way? Not OpenPhil.
Eliezer’s main point of his ~20k words isn’t really what I want to defend, but I will say a few words about how I would constructively interpret it. I think Eliezer’s main claim is that he has an intuitive system-1 model of the inhomogeneous Poisson process that emits working AGI systems, and that this model isn’t informed by the compute equivalents of biological anchors, and that his lack of being informed by that isn’t a mistake. I’m not sure if he’s actually making the stronger claim that anyone whose model is informed by the biological anchors is making a mistake, but if so, I don’t agree. My own model is somewhat informed by biological anchors; it’s more informed by what the TAI report calls “subjective impressiveness extrapolation”, extrapolations on benchmark performance, and some vague sense of other point processes that emit AI winters and major breakthroughs. Someone who has total Knightean uncertainty and no intuitive models of how AGI comes about would surely do well to adopt OpenPhil’s distribution as a prior.
(I’m not sure whether your summary captures Eliezer’s view, but strong-upvoted for what strikes me as a reasonable attempt.)
I suppose this kind of report is less useful to you the more you think the uncertainty lies in the compute-efficiency translation factor variable. If you think most of the orders of magnitude are in that value, you don’t care so much about the biological anchors.
And maybe you’re in that state if you think building AGI is just a matter of coming up with clever algorithms. But if you think there’s just some as-yet undiscovered general reasoning algorithm, and that’s really the only thing that matters for AGI, then why are you at all impressed by increasingly capable AI systems that use more and more compute, like AlphaGo or GPT-3? It’s (supposedly) not general reasoning, so why does it matter?
It seems to me like the compute-efficiency translation factor is just a perfectly reasonable non-mysterious quantity that we can also get information about and estimate. It’s not going to be the same factor across all tasks, but it seems like we could at least get some idea of what quantities it plausibly could take on by looking at how much compute current (and old) systems are using to match human performance for various tasks, and looking at how that number varies across tasks and changes over time.
I wouldn’t expect such analysis to leave the translation factor so uncertain that our total uncertainty is concentrated so overwhelmingly in that parameter that the biological anchors become useless.