But it turns out that the technology didn’t just happen to be just about to be discovered in any case. Instead, there was a direct connection between the prediction and its undoing. In my view, that makes the prediction less “surprisingly” incorrect.[2]
? If you are trying to make the point that technology is unpredictable, an example of a ‘direct connection’ and backfiring is a great example because it shows how fundamentally unpredictable things are: he could hardly have expected that his dismissal would spur an epochal discovery and that seems extremely surprising; this supports Ord & Yudkowsky, it doesn’t contradict them. And if you’re trying to make a claim that forecasts systematically backfire, that’s even more alarming than O/Y’s claims, because it means that expert forecasts will not just make a nontrivial number of errors (enough to be an x-risk concern) but will be systematically inversely correlated with risks and the biggest risks will come from the ones experts most certainly forecast to not be risks...
But the footnote also suggests to me that this may not have been a failure of forecasting at all, or only a minor one. Hearing that Fermi thought that something that ended up happening was only a “remote possibility” seems to suggest he was wildly off the mark. But if he actually thought the chance was 10%, perhaps he was “right” in some sense—e.g., perhaps he was well-calibrated—and this just happened to be one of the 1 in 10 times that a 10% likely outcome occurs.
So to summarize that case study criticism: everything you factchecked was accurate and you have no evidence of any kind that the Fermi story does not mean what O/Y interpret it as.
Furthermore, even if that was a genuine prediction Wright made at the time, it seems it was a prediction made briefly, once, during many years of working on a topic, and which wasn’t communicated publicly. Thus, even if it was a genuine prediction, it may have little bearing on the trustworthiness in general of publicly made forecasts about technological developments.
So to summarize that case study criticism: everything you factchecked was accurate and you have no evidence of any kind that the Wright story does not mean what O/Y interpret it as.
Let’s imagine that all of my above points turn out to be unfounded or unimportant
Of the 4 case studies you criticize, your claim actually supports them in the first one, you agree the second one is accurate, and you provide only speculations and no actual criticisms in the third and fourth.
So to summarize that case study criticism: everything you factchecked was accurate and you have no evidence of any kind that the Fermi story does not mean what O/Y interpret it as.
Re: the Fermi quote, this does not seem to be an accurate summary to me. Learning that Fermi meant 10% when he said “remote possibility” does in fact change how I view that incident.
If it were common knowledge that any hyperbolic language experts use when speaking about the unlikelihood of AGI (e.g. Andrew Ng’s statement “worrying about AI safety is like worrying about overpopulation on Mars”) actually corresponded to a 10% subjective probability of AGI, things would look very different than they currently do.
More generally, on a strategic level there is very little difference between a genuinely incorrect forecast and one that is “correct”, but communicated so poorly as to create a wrong impression in the mind of the listener. If the state of affairs is such that anyone who privately believes there is a 10% chance of AGI is incentivized to instead report their assessment as “remote”, the conclusion of Ord/Yudkowsky holds, and it remains impossible to discern whether AGI is imminent by listening to expert forecasts.
(I also don’t believe that said experts, if asked to translate their forecasts to numerical probabilities, would give a median estimate anywhere near as high as 10%, but that’s largely tangential to the discussion at hand.)
Furthermore, and more importantly, however: I deny that Fermi’s 10% somehow detracts from the point that forecasting the future of novel technologies is hard.
Four years prior to overseeing the world’s first nuclear reaction, Fermi believed that it was more likely than not that a nuclear chain reaction was impossible. Setting aside for a moment the question of whether Fermi’s specific probability assignment was negligible, or merely small, what this indicates is that the majority of the information necessary to determine the possibility of a nuclear chain reaction was in fact unavailable to Fermi at the time he made his forecast. This does not support the idea that making predictions about technology is easy, any more than it would have if Fermi had assigned 0.001% instead of 10%!
More generally, the specific probability estimate Fermi gave is nothing more than a red herring, one that is given undue attention by the OP. The relevant factor to Ord/Yudkowsky’s thesis is how much uncertainty there is in the probability distribution of a given technology—not whether the mean of said distribution, when treated as a point estimate, happens to be negligible or non-negligible. Focusing too much on the latter not only obfuscates the correct lesson to be learned, but also sometimes leads to nonsensical results.
If it were common knowledge that any hyperbolic language experts use when speaking about the unlikelihood of AGI (e.g. Andrew Ng’s statement “worrying about AI safety is like worrying about overpopulation on Mars”) actually corresponded to a 10% subjective probability of AGI, things would look very different than they currently do.
Did you have anything specific in mind about how things would look different? I have the impression that you’re trying to imply something in particular, but I’m not sure what it is.
EDIT: Also, I’m a little confused about whether you mean to be agreeing with me or disagreeing. The tone of your comment sounds like disagreeing, but content-wise it seems like we’re both agreeing that if someone is using language like “remote possibility” to mean 10%, that is a noteworthy and not-generally-obvious fact.
Maybe you’re saying that experts do frequently obfuscate with hyperbolic language, s.t. it’s not surprising to you that Fermi would mean 10% when he said “remote possibility”, but that this fact is not generally recognized. (And things would look very different if it was.) Is that it?
I think this comment raises some valid and interesting points. But I’d push back a bit on some points.
(Note that this comment was written quickly, so I may say things a bit unclearly or be saying opinions I haven’t mulled over for a long time.)
More generally, on a strategic level there is very little difference between a genuinely incorrect forecast and one that is “correct”, but communicated so poorly as to create a wrong impression in the mind of the listener.
There’s at least some truth to this. But it’s also possible to ask experts to give a number, as Fermi was asked. If the problem is poor communication, then asking experts to give a number will resolve at least part of the problem (though substantial damage may have been done by planting the verbal estimate in people’s mind). If the problem is poor estimation, then asking for an explicit estimate might make things worse, as it could give a more precise incorrect answer for people to anchor on. (I don’t know of specific evidence that people anchor more on numerical than verbal probability statements, but it seems likely me. Also, to be clear, despite this, I think I’m generally in favour of explicit probability estimates in many cases.)
If the state of affairs is such that anyone who privately believes there is a 10% chance of AGI is incentivized to instead report their assessment as “remote”, the conclusion of Ord/Yudkowsky holds, and it remains impossible to discern whether AGI is imminent by listening to expert forecasts.
I think this is true if no one asks the experts for explicit numerical estimate, or if the incentives to avoid giving such estimates are strong enough that experts will refuse when asked. I think both of those conditions hold to a substantial extent in the real world and in relation to AGI, and that that is a reason why the Fermi case has substantial relevance to the AGI case. But it still seems useful to me to be aware of the distinction between failures of communication vs of estimation, as it seems we could sometimes get evidence that discriminates between which of those is occurring/common, and that which is occurring/common could sometimes be relevant.
Furthermore, and more importantly, however: I deny that Fermi’s 10% somehow detracts from the point that forecasting the future of novel technologies is hard.
I definitely wasn’t claiming that forecasting the future of novel technologies is easy, and I didn’t interpret ESRogs as doing so either. What I was exploring was merely whether this case is a clear case of an expert’s technology forecast being “wrong” (and, if so, “how wrong”), and what this reflects about the typical accuracy of expert technology forecasts. They could conceivably be typically accurate even if very very hard to make, if experts are really good at it and put in lots of effort. But I think more likely they’re often wrong. The important question is essentially “how often”, and this post bites off the smaller question “what does the Fermi case tell us about that”.
As for the rest of the comment, I think both the point estimates and the uncertainty are relevant, at least when judging estimates (rather than making decisions based on them). This is in line with my understanding from e.g. Tetlock’s work. I don’t think I’d read much into an expert saying 1% rather than 10% for something as hard to forecast as an unprecedented tech development, unless I had reason to believe the expert was decently calibrated. But if they have given one of those numbers, and then we see what happens, then which number they gave makes a difference to how calibrated vs uncalibrated I should see them as (which I might then generalise in a weak way to experts more widely).
That said, I do generally think uncertainty of estimates is very important, and think the paper you linked to makes that point very well. And I do think one could easily focus too much on point estimates; e.g., I wouldn’t plug Ord’s existential risk estimates into a model as point estimates without explicitly representing a lot of uncertainty too.
? If you are trying to make the point that technology is unpredictable, an example of a ‘direct connection’ and backfiring is a great example because it shows how fundamentally unpredictable things are: he could hardly have expected that his dismissal would spur an epochal discovery and that seems extremely surprising; this supports Ord & Yudkowsky, it doesn’t contradict them. And if you’re trying to make a claim that forecasts systematically backfire, that’s even more alarming than O/Y’s claims, because it means that expert forecasts will not just make a nontrivial number of errors (enough to be an x-risk concern) but will be systematically inversely correlated with risks and the biggest risks will come from the ones experts most certainly forecast to not be risks...
I think this paragraph makes valid points, and have updated in response (as well as in response to ESRogs indication of agreement). Here are my updated thoughts on the relevance of the “direct connection”:
I may be wrong about the “direct connection” slightly weakening the evidence this case provides for Ord and Yudkowsky’s claims. I still feel like there’s something to that, but I find it hard to explain it precisely, and I’ll take that, plus the responses from you and ESRogs, as evidence that there’s less going on here than I think.
I guess I’d at least stand by my literal phrasings in that section, which were just about my perceptions. But perhaps those perceptions were erroneous or idiosyncratic, and perhaps to the point where they weren’t worth raising.
That said, it also seems possible to me that, even if there’s no “real” reason why a lack of direction connection should make this more “surprising”, many people would (like me) erroneously feel it does. This could perhaps be why Ord writes “the very next morning” rather than just “the next morning”.
Perhaps what I should’ve emphasised more is the point I make in footnote 2 (which is also in line with some of what you say):
This may not reduce the strength of the evidence this case provides for certain claims. One such claim would be that we should put little trust in experts’ forecasts of AGI being definitely a long way off, and this is specifically because such forecasts may themselves annoy other researchers and spur them to develop AGI faster. But Ord and Yudkowsky didn’t seem to be explicitly making claims like that.
Interestingly, Yudkowsky makes similar point in the essay this post partially responds to: “(Also, Demis Hassabis was present, so [people at a conference who were asked to make a particular forecast] all knew that if they named something insufficiently impossible, Demis would have DeepMind go and do it [and thereby make their forecast inaccurate].)” (Also, again, as I noted in this post, I do like that essay.)
I think that that phenomenon would cause some negative correlation between forecasts and truth, in some cases. I expect that, for the most part, that’d get largely overwhelmed by a mixture of random inaccuracies and a weak tendency towards accuracy. I wouldn’t claim that, overall, “forecasts systematically backfire”.
So to summarize that case study criticism: everything you factchecked was accurate and you have no evidence of any kind that the Fermi story does not mean what O/Y interpret it as.
I find this a slightly odd sentence. My “fact-check” was literally just quoting and thinking about Ord’s own footnote. So it would be very odd if that resulted in discovering that Ord was inaccurate. This connects back to the point I make in my first comment response: this post was not a takedown.
My point here was essentially that:
I think the main text of Ord’s book (without the footnote) would make a reader think Fermi’s forecast was very very wrong.
But in reality it is probably better interpreted as very very poorly communicated (which is itself relevant and interesting), and either somewhat wrong or well-calibrated but unlucky.
I do think the vast majority of people would think “remote possibility” means far less than 10%.
Firstly, I think I should say that this post was very much not intended as anything like a scathing takedown of Ord and Yudkowsky’s claims or evidence. Nor did I mean to imply I’m giving definitive arguments that these cases provide no evidence for the claims made. I mean this to have more of a collaborative than combative spirit in relation to Ord and Yudkowsky’s projects.
My aim was simply to “prod at each suspicious plank on its own terms, and update incrementally.” And my key conclusion is that the authors, “in my opinion, imply these cases support their claims more clearly than they do”—not that the cases provide no evidence. It seems to me healthy to question evidence we have—even for conclusions we do still think are right, and even when our questions don’t definitively cut down the evidence, but rather raise reasons for some doubt.
It’s possible I could’ve communicated that better, and I’m open to suggestions on that front. But from re-reading the post again, especially the intro and conclusion, it does seem I repeatedly made explicit statements to this effect. (Although I did realise after going to bed last night that the “And I don’t think we should update much...” sentence was off, so I’ve now made that a tad clearer.)
I’ve split my response about the Rutherford and Fermi cases into different comments.
Of the 4 case studies you criticize, your claim actually supports them in the first one, you agree the second one is accurate, and you provide only speculations and no actual criticisms in the third and fourth.
Again, I think this sentence may reflect interpreting this post as much more strident and critical than it was really meant to be. I may be wrong about the “direct connection” thing (discussed in a separate comment), but I do think I raise plausible reasons for at least some doubt about (rather than outright dismissal of) the evidence each case provides, compared to how a reader might initially interpret them.
I’m also not sure what “only speculations and no actual criticisms” would mean. If you mean e.g. that I don’t have evidence that a lot of Americans would’ve believed nuclear weapons would exist someday, then yes, that’s true. I don’t claim otherwise. But I point out a potentially relevant disanalogy between nuclear weapons development and AI development. And I point give some evidence that “the group of people who did know about nuclear weapons before the bombing of Hiroshima, or who believed such weapons may be developed soon, was (somewhat) larger than one might think from reading Yudkowsky’s essay.” And I do give some evidence for that, as well as pointing out that I’m not aware of evidence either way for one relevant point.
Also, I don’t really claim any of this post to be “criticism”, at least in the usual fairly negative sense, just “prod[ding] at each suspicious plank on its own terms”. I’m explicitly intending to make only relatively weak claims, really.
And then the “Sample size and representativeness” section provides largely separate reasons why it might not make much sense to update much on these cases (at least from a relatively moderate starting point) even ignoring those reasons for doubt. (Though see the interesting point 3 in Daniel Kokotajlo’s comment.)
Of the 4 case studies you criticize, your claim actually supports them in the first one, you agree the second one is accurate
I agree with you about which way the direct connection points. But I think the point about Rutherford’s potential deliberate obfuscation is significant.
? If you are trying to make the point that technology is unpredictable, an example of a ‘direct connection’ and backfiring is a great example because it shows how fundamentally unpredictable things are: he could hardly have expected that his dismissal would spur an epochal discovery and that seems extremely surprising; this supports Ord & Yudkowsky, it doesn’t contradict them. And if you’re trying to make a claim that forecasts systematically backfire, that’s even more alarming than O/Y’s claims, because it means that expert forecasts will not just make a nontrivial number of errors (enough to be an x-risk concern) but will be systematically inversely correlated with risks and the biggest risks will come from the ones experts most certainly forecast to not be risks...
So to summarize that case study criticism: everything you factchecked was accurate and you have no evidence of any kind that the Fermi story does not mean what O/Y interpret it as.
So to summarize that case study criticism: everything you factchecked was accurate and you have no evidence of any kind that the Wright story does not mean what O/Y interpret it as.
Of the 4 case studies you criticize, your claim actually supports them in the first one, you agree the second one is accurate, and you provide only speculations and no actual criticisms in the third and fourth.
Re: the Fermi quote, this does not seem to be an accurate summary to me. Learning that Fermi meant 10% when he said “remote possibility” does in fact change how I view that incident.
If it were common knowledge that any hyperbolic language experts use when speaking about the unlikelihood of AGI (e.g. Andrew Ng’s statement “worrying about AI safety is like worrying about overpopulation on Mars”) actually corresponded to a 10% subjective probability of AGI, things would look very different than they currently do.
More generally, on a strategic level there is very little difference between a genuinely incorrect forecast and one that is “correct”, but communicated so poorly as to create a wrong impression in the mind of the listener. If the state of affairs is such that anyone who privately believes there is a 10% chance of AGI is incentivized to instead report their assessment as “remote”, the conclusion of Ord/Yudkowsky holds, and it remains impossible to discern whether AGI is imminent by listening to expert forecasts.
(I also don’t believe that said experts, if asked to translate their forecasts to numerical probabilities, would give a median estimate anywhere near as high as 10%, but that’s largely tangential to the discussion at hand.)
Furthermore, and more importantly, however: I deny that Fermi’s 10% somehow detracts from the point that forecasting the future of novel technologies is hard.
Four years prior to overseeing the world’s first nuclear reaction, Fermi believed that it was more likely than not that a nuclear chain reaction was impossible. Setting aside for a moment the question of whether Fermi’s specific probability assignment was negligible, or merely small, what this indicates is that the majority of the information necessary to determine the possibility of a nuclear chain reaction was in fact unavailable to Fermi at the time he made his forecast. This does not support the idea that making predictions about technology is easy, any more than it would have if Fermi had assigned 0.001% instead of 10%!
More generally, the specific probability estimate Fermi gave is nothing more than a red herring, one that is given undue attention by the OP. The relevant factor to Ord/Yudkowsky’s thesis is how much uncertainty there is in the probability distribution of a given technology—not whether the mean of said distribution, when treated as a point estimate, happens to be negligible or non-negligible. Focusing too much on the latter not only obfuscates the correct lesson to be learned, but also sometimes leads to nonsensical results.
Did you have anything specific in mind about how things would look different? I have the impression that you’re trying to imply something in particular, but I’m not sure what it is.
EDIT: Also, I’m a little confused about whether you mean to be agreeing with me or disagreeing. The tone of your comment sounds like disagreeing, but content-wise it seems like we’re both agreeing that if someone is using language like “remote possibility” to mean 10%, that is a noteworthy and not-generally-obvious fact.
Maybe you’re saying that experts do frequently obfuscate with hyperbolic language, s.t. it’s not surprising to you that Fermi would mean 10% when he said “remote possibility”, but that this fact is not generally recognized. (And things would look very different if it was.) Is that it?
Minor thing: did you mean to refer to Fermi rather than to Rutherford in that last paragraph?
Oops, yes. Fixed.
I think this comment raises some valid and interesting points. But I’d push back a bit on some points.
(Note that this comment was written quickly, so I may say things a bit unclearly or be saying opinions I haven’t mulled over for a long time.)
There’s at least some truth to this. But it’s also possible to ask experts to give a number, as Fermi was asked. If the problem is poor communication, then asking experts to give a number will resolve at least part of the problem (though substantial damage may have been done by planting the verbal estimate in people’s mind). If the problem is poor estimation, then asking for an explicit estimate might make things worse, as it could give a more precise incorrect answer for people to anchor on. (I don’t know of specific evidence that people anchor more on numerical than verbal probability statements, but it seems likely me. Also, to be clear, despite this, I think I’m generally in favour of explicit probability estimates in many cases.)
I think this is true if no one asks the experts for explicit numerical estimate, or if the incentives to avoid giving such estimates are strong enough that experts will refuse when asked. I think both of those conditions hold to a substantial extent in the real world and in relation to AGI, and that that is a reason why the Fermi case has substantial relevance to the AGI case. But it still seems useful to me to be aware of the distinction between failures of communication vs of estimation, as it seems we could sometimes get evidence that discriminates between which of those is occurring/common, and that which is occurring/common could sometimes be relevant.
I definitely wasn’t claiming that forecasting the future of novel technologies is easy, and I didn’t interpret ESRogs as doing so either. What I was exploring was merely whether this case is a clear case of an expert’s technology forecast being “wrong” (and, if so, “how wrong”), and what this reflects about the typical accuracy of expert technology forecasts. They could conceivably be typically accurate even if very very hard to make, if experts are really good at it and put in lots of effort. But I think more likely they’re often wrong. The important question is essentially “how often”, and this post bites off the smaller question “what does the Fermi case tell us about that”.
As for the rest of the comment, I think both the point estimates and the uncertainty are relevant, at least when judging estimates (rather than making decisions based on them). This is in line with my understanding from e.g. Tetlock’s work. I don’t think I’d read much into an expert saying 1% rather than 10% for something as hard to forecast as an unprecedented tech development, unless I had reason to believe the expert was decently calibrated. But if they have given one of those numbers, and then we see what happens, then which number they gave makes a difference to how calibrated vs uncalibrated I should see them as (which I might then generalise in a weak way to experts more widely).
That said, I do generally think uncertainty of estimates is very important, and think the paper you linked to makes that point very well. And I do think one could easily focus too much on point estimates; e.g., I wouldn’t plug Ord’s existential risk estimates into a model as point estimates without explicitly representing a lot of uncertainty too.
I think this paragraph makes valid points, and have updated in response (as well as in response to ESRogs indication of agreement). Here are my updated thoughts on the relevance of the “direct connection”:
I may be wrong about the “direct connection” slightly weakening the evidence this case provides for Ord and Yudkowsky’s claims. I still feel like there’s something to that, but I find it hard to explain it precisely, and I’ll take that, plus the responses from you and ESRogs, as evidence that there’s less going on here than I think.
I guess I’d at least stand by my literal phrasings in that section, which were just about my perceptions. But perhaps those perceptions were erroneous or idiosyncratic, and perhaps to the point where they weren’t worth raising.
That said, it also seems possible to me that, even if there’s no “real” reason why a lack of direction connection should make this more “surprising”, many people would (like me) erroneously feel it does. This could perhaps be why Ord writes “the very next morning” rather than just “the next morning”.
Perhaps what I should’ve emphasised more is the point I make in footnote 2 (which is also in line with some of what you say):
Interestingly, Yudkowsky makes similar point in the essay this post partially responds to: “(Also, Demis Hassabis was present, so [people at a conference who were asked to make a particular forecast] all knew that if they named something insufficiently impossible, Demis would have DeepMind go and do it [and thereby make their forecast inaccurate].)” (Also, again, as I noted in this post, I do like that essay.)
I think that that phenomenon would cause some negative correlation between forecasts and truth, in some cases. I expect that, for the most part, that’d get largely overwhelmed by a mixture of random inaccuracies and a weak tendency towards accuracy. I wouldn’t claim that, overall, “forecasts systematically backfire”.
I find this a slightly odd sentence. My “fact-check” was literally just quoting and thinking about Ord’s own footnote. So it would be very odd if that resulted in discovering that Ord was inaccurate. This connects back to the point I make in my first comment response: this post was not a takedown.
My point here was essentially that:
I think the main text of Ord’s book (without the footnote) would make a reader think Fermi’s forecast was very very wrong.
But in reality it is probably better interpreted as very very poorly communicated (which is itself relevant and interesting), and either somewhat wrong or well-calibrated but unlucky.
I do think the vast majority of people would think “remote possibility” means far less than 10%.
Firstly, I think I should say that this post was very much not intended as anything like a scathing takedown of Ord and Yudkowsky’s claims or evidence. Nor did I mean to imply I’m giving definitive arguments that these cases provide no evidence for the claims made. I mean this to have more of a collaborative than combative spirit in relation to Ord and Yudkowsky’s projects.
My aim was simply to “prod at each suspicious plank on its own terms, and update incrementally.” And my key conclusion is that the authors, “in my opinion, imply these cases support their claims more clearly than they do”—not that the cases provide no evidence. It seems to me healthy to question evidence we have—even for conclusions we do still think are right, and even when our questions don’t definitively cut down the evidence, but rather raise reasons for some doubt.
It’s possible I could’ve communicated that better, and I’m open to suggestions on that front. But from re-reading the post again, especially the intro and conclusion, it does seem I repeatedly made explicit statements to this effect. (Although I did realise after going to bed last night that the “And I don’t think we should update much...” sentence was off, so I’ve now made that a tad clearer.)
I’ve split my response about the Rutherford and Fermi cases into different comments.
Again, I think this sentence may reflect interpreting this post as much more strident and critical than it was really meant to be. I may be wrong about the “direct connection” thing (discussed in a separate comment), but I do think I raise plausible reasons for at least some doubt about (rather than outright dismissal of) the evidence each case provides, compared to how a reader might initially interpret them.
I’m also not sure what “only speculations and no actual criticisms” would mean. If you mean e.g. that I don’t have evidence that a lot of Americans would’ve believed nuclear weapons would exist someday, then yes, that’s true. I don’t claim otherwise. But I point out a potentially relevant disanalogy between nuclear weapons development and AI development. And I point give some evidence that “the group of people who did know about nuclear weapons before the bombing of Hiroshima, or who believed such weapons may be developed soon, was (somewhat) larger than one might think from reading Yudkowsky’s essay.” And I do give some evidence for that, as well as pointing out that I’m not aware of evidence either way for one relevant point.
Also, I don’t really claim any of this post to be “criticism”, at least in the usual fairly negative sense, just “prod[ding] at each suspicious plank on its own terms”. I’m explicitly intending to make only relatively weak claims, really.
And then the “Sample size and representativeness” section provides largely separate reasons why it might not make much sense to update much on these cases (at least from a relatively moderate starting point) even ignoring those reasons for doubt. (Though see the interesting point 3 in Daniel Kokotajlo’s comment.)
I agree with you about which way the direct connection points. But I think the point about Rutherford’s potential deliberate obfuscation is significant.