It seems to me that it’s possible to simultaneously hold that (a) “fire alarms” do not naturally occur for a supermajority of real-life risks, and (b) that this is in fact quite bad, because a mechanism for creating common knowledge is never established.
Your essay seems to mostly argue in favor of (a), but I don’t see this as a refutation (or even a counterweight) to Eliezer’s point; if anything, I expect he would agree with you that naturally-occurring fire alarms are quite uncommon! Common-knowledge-creation mechanisms are costly to establish; we shouldn’t expect them to show up in situations where no capital was invested in establishing them.
(b), on the other hand, seems to me like the load-bearing part of the argument! The evidence that people coordinate well in advance of a risk (as opposed to after it), in the absence of any kind of common-knowledge-creation mechanism, seems remarkably slim to me. On the other hand, there seems to be quite some evidence against this proposition, with the most obvious counterexample at the moment being the international response to COVID-19. I don’t see how, on your thesis, the COVID-19 pandemic became as bad as it did; or (for that matter) how the Challenger disaster occurred, how the Chernobyl reactor meltdown occurred, or why global warming is still ongoing.
This seems to me like it empirically refutes your whole thesis, as well as any implications it may have for AI safety. What am I missing here?
It seems to me that it’s possible to simultaneously hold that (a) “fire alarms” do not naturally occur for a supermajority of real-life risks, and (b) that this is in fact quite bad, because a mechanism for creating common knowledge is never established.
Your essay seems to mostly argue in favor of (a), but I don’t see this as a refutation (or even a counterweight) to Eliezer’s point; if anything, I expect he would agree with you that naturally-occurring fire alarms are quite uncommon! Common-knowledge-creation mechanisms are costly to establish; we shouldn’t expect them to show up in situations where no capital was invested in establishing them.
(b), on the other hand, seems to me like the load-bearing part of the argument! The evidence that people coordinate well in advance of a risk (as opposed to after it), in the absence of any kind of common-knowledge-creation mechanism, seems remarkably slim to me. On the other hand, there seems to be quite some evidence against this proposition, with the most obvious counterexample at the moment being the international response to COVID-19. I don’t see how, on your thesis, the COVID-19 pandemic became as bad as it did; or (for that matter) how the Challenger disaster occurred, how the Chernobyl reactor meltdown occurred, or why global warming is still ongoing.
This seems to me like it empirically refutes your whole thesis, as well as any implications it may have for AI safety. What am I missing here?