? If you are trying to make the point that technology is unpredictable, an example of a ‘direct connection’ and backfiring is a great example because it shows how fundamentally unpredictable things are: he could hardly have expected that his dismissal would spur an epochal discovery and that seems extremely surprising; this supports Ord & Yudkowsky, it doesn’t contradict them. And if you’re trying to make a claim that forecasts systematically backfire, that’s even more alarming than O/Y’s claims, because it means that expert forecasts will not just make a nontrivial number of errors (enough to be an x-risk concern) but will be systematically inversely correlated with risks and the biggest risks will come from the ones experts most certainly forecast to not be risks...
I think this paragraph makes valid points, and have updated in response (as well as in response to ESRogs indication of agreement). Here are my updated thoughts on the relevance of the “direct connection”:
I may be wrong about the “direct connection” slightly weakening the evidence this case provides for Ord and Yudkowsky’s claims. I still feel like there’s something to that, but I find it hard to explain it precisely, and I’ll take that, plus the responses from you and ESRogs, as evidence that there’s less going on here than I think.
I guess I’d at least stand by my literal phrasings in that section, which were just about my perceptions. But perhaps those perceptions were erroneous or idiosyncratic, and perhaps to the point where they weren’t worth raising.
That said, it also seems possible to me that, even if there’s no “real” reason why a lack of direction connection should make this more “surprising”, many people would (like me) erroneously feel it does. This could perhaps be why Ord writes “the very next morning” rather than just “the next morning”.
Perhaps what I should’ve emphasised more is the point I make in footnote 2 (which is also in line with some of what you say):
This may not reduce the strength of the evidence this case provides for certain claims. One such claim would be that we should put little trust in experts’ forecasts of AGI being definitely a long way off, and this is specifically because such forecasts may themselves annoy other researchers and spur them to develop AGI faster. But Ord and Yudkowsky didn’t seem to be explicitly making claims like that.
Interestingly, Yudkowsky makes similar point in the essay this post partially responds to: “(Also, Demis Hassabis was present, so [people at a conference who were asked to make a particular forecast] all knew that if they named something insufficiently impossible, Demis would have DeepMind go and do it [and thereby make their forecast inaccurate].)” (Also, again, as I noted in this post, I do like that essay.)
I think that that phenomenon would cause some negative correlation between forecasts and truth, in some cases. I expect that, for the most part, that’d get largely overwhelmed by a mixture of random inaccuracies and a weak tendency towards accuracy. I wouldn’t claim that, overall, “forecasts systematically backfire”.
I think this paragraph makes valid points, and have updated in response (as well as in response to ESRogs indication of agreement). Here are my updated thoughts on the relevance of the “direct connection”:
I may be wrong about the “direct connection” slightly weakening the evidence this case provides for Ord and Yudkowsky’s claims. I still feel like there’s something to that, but I find it hard to explain it precisely, and I’ll take that, plus the responses from you and ESRogs, as evidence that there’s less going on here than I think.
I guess I’d at least stand by my literal phrasings in that section, which were just about my perceptions. But perhaps those perceptions were erroneous or idiosyncratic, and perhaps to the point where they weren’t worth raising.
That said, it also seems possible to me that, even if there’s no “real” reason why a lack of direction connection should make this more “surprising”, many people would (like me) erroneously feel it does. This could perhaps be why Ord writes “the very next morning” rather than just “the next morning”.
Perhaps what I should’ve emphasised more is the point I make in footnote 2 (which is also in line with some of what you say):
Interestingly, Yudkowsky makes similar point in the essay this post partially responds to: “(Also, Demis Hassabis was present, so [people at a conference who were asked to make a particular forecast] all knew that if they named something insufficiently impossible, Demis would have DeepMind go and do it [and thereby make their forecast inaccurate].)” (Also, again, as I noted in this post, I do like that essay.)
I think that that phenomenon would cause some negative correlation between forecasts and truth, in some cases. I expect that, for the most part, that’d get largely overwhelmed by a mixture of random inaccuracies and a weak tendency towards accuracy. I wouldn’t claim that, overall, “forecasts systematically backfire”.