Example chosen because (a) I have really, absolutely no idea what the optimal action is here; (b) I have reason to belive that the risk is kind of low, anyway. (c) most commentors here wont have a strong opinion, so we wont have a flame war over what the right answer might be. Let it stand in for other cases where it is very, very unclear what the optimal action is.
This analogy might not work for all the things “dragons” is standing in for in this thread … but if I have a good statistical bound on the risk posed by dragons being low (but cannot, strictly speaking, rule out their existence entirely) I may conclude that a residual 1E-5 chance of running in to one to be a acceptable risk.
So if I see verified reports of AI causing a mass casualty incident with more that %500 million in damage (or whatever the threshold in the California bill is), I shall consider that evidence on a par to seeing Lake-Town get toasted by Smaug, and update accordingly.
Example chosen because (a) I have really, absolutely no idea what the optimal action is here; (b) I have reason to belive that the risk is kind of low, anyway. (c) most commentors here wont have a strong opinion, so we wont have a flame war over what the right answer might be. Let it stand in for other cases where it is very, very unclear what the optimal action is.
This analogy might not work for all the things “dragons” is standing in for in this thread … but if I have a good statistical bound on the risk posed by dragons being low (but cannot, strictly speaking, rule out their existence entirely) I may conclude that a residual 1E-5 chance of running in to one to be a acceptable risk.
So if I see verified reports of AI causing a mass casualty incident with more that %500 million in damage (or whatever the threshold in the California bill is), I shall consider that evidence on a par to seeing Lake-Town get toasted by Smaug, and update accordingly.