I think the most important concept here is Normalization of Deviance. I think this is the default and expected outcome if an organization tries to import high-reliability practices that are very expensive, that don’t fit the problem domain, or that lack the ability to simulate failures. Reporting culture/just culture seems correct and important, and there are a few other interesting lessons, but most stuff is going to fail to transfer.
Most lessons from aviation and medicine won’t transfer, because these are well-understood domains where it’s feasible to use checklists, and it’s reasonable to expect investing in the checklists to help. AI research is mostly not like that. A big risk I see is that people will try to transfer practices that don’t fit, and that this will serve to drive symbolic ineffectual actions plus normalization of deviance.
That said, there are a few interesting small lessons that I picked up from watching aviation accident analysis videos, mostly under the label “crew resource management”. Commercial airliners are overstaffed to handle the cases where workload spikes suddenly, or someone is screwing up and needs someone else to notice, or someone becomes incapacitated at a key moment. One recurring failure theme in incidents is excessive power gradients within a flight crew: situations where the captain is f*ing up, but the other crewmembers are too timid to point it out. Ie, deference to expertise is usually presented as a failure mode. (This may be an artifact of there being a high minimum bar for the skill level of people in a cockpit, which software companies fail to uphold.)
(Note that I haven’t said anything about nuclear, because I think managing nuclear power plants is actually just an easy problem. That is, I think nearly all the safety strategies surrounding nuclear power are symbolic, unnecessary, and motivated by unfounded paranoia.)
(I also haven’t said much about medicine because I think hospitals are, in practice, pretty clearly not high-reliability in the ways that matter. Ie, they may have managed to drive the rates of a few specific legible errors down to near zero, but the overall error rate is still high; it would be a mistake to hold up surgical-infection-rate as a sign that hospitals are high-reliability orgs, and not also observe that they’re pretty bad at a lot of other things.)
(Note that I haven’t said anything about nuclear, because I think managing nuclear power plants is actually just an easy problem. That is, I think nearly all the safety strategies surrounding nuclear power are symbolic, unnecessary, and motivated by unfounded paranoia.)
I think managing a nuclear power plant is relatively easy… but, like, I’d expect managing a regular ol’ power plant (or, many other industries) to be pretty easy, and my impression is that random screwups still manage to happen. So it still seems notable to me if nuclear plants avoid the same degree of incidents that other random utility companies have.
I agree many of the safety strategies surrounding nuclear power are symbolic. But I’ve recently updated away from “yay nuclear power” because while it seems to me the current generation of stuff is just pretty safe and is being sort of unreasonably strangled by regulation… I dunno that I actually expect civilizational competence to continue having it be safe, given how I’ve seen the rest of civilizational competence playing otu. (I’m not very informed here though)
I think the most important concept here is Normalization of Deviance. I think this is the default and expected outcome if an organization tries to import high-reliability practices that are very expensive, that don’t fit the problem domain, or that lack the ability to simulate failures. Reporting culture/just culture seems correct and important, and there are a few other interesting lessons, but most stuff is going to fail to transfer.
Most lessons from aviation and medicine won’t transfer, because these are well-understood domains where it’s feasible to use checklists, and it’s reasonable to expect investing in the checklists to help. AI research is mostly not like that. A big risk I see is that people will try to transfer practices that don’t fit, and that this will serve to drive symbolic ineffectual actions plus normalization of deviance.
That said, there are a few interesting small lessons that I picked up from watching aviation accident analysis videos, mostly under the label “crew resource management”. Commercial airliners are overstaffed to handle the cases where workload spikes suddenly, or someone is screwing up and needs someone else to notice, or someone becomes incapacitated at a key moment. One recurring failure theme in incidents is excessive power gradients within a flight crew: situations where the captain is f*ing up, but the other crewmembers are too timid to point it out. Ie, deference to expertise is usually presented as a failure mode. (This may be an artifact of there being a high minimum bar for the skill level of people in a cockpit, which software companies fail to uphold.)
(Note that I haven’t said anything about nuclear, because I think managing nuclear power plants is actually just an easy problem. That is, I think nearly all the safety strategies surrounding nuclear power are symbolic, unnecessary, and motivated by unfounded paranoia.)
(I also haven’t said much about medicine because I think hospitals are, in practice, pretty clearly not high-reliability in the ways that matter. Ie, they may have managed to drive the rates of a few specific legible errors down to near zero, but the overall error rate is still high; it would be a mistake to hold up surgical-infection-rate as a sign that hospitals are high-reliability orgs, and not also observe that they’re pretty bad at a lot of other things.)
More on the normalization of deviance: Challenger Launch Decision by Diane Vaughan (book outline).
I think managing a nuclear power plant is relatively easy… but, like, I’d expect managing a regular ol’ power plant (or, many other industries) to be pretty easy, and my impression is that random screwups still manage to happen. So it still seems notable to me if nuclear plants avoid the same degree of incidents that other random utility companies have.
I agree many of the safety strategies surrounding nuclear power are symbolic. But I’ve recently updated away from “yay nuclear power” because while it seems to me the current generation of stuff is just pretty safe and is being sort of unreasonably strangled by regulation… I dunno that I actually expect civilizational competence to continue having it be safe, given how I’ve seen the rest of civilizational competence playing otu. (I’m not very informed here though)