(Disclaimer: I’m not very knowledgeable about safety engineering or formal proofs)
I notice that whenever someone brings up “What if this unexpected thing happens?”, you emphasize that it’s about not causing accidents. I’m worried that it’s hard to define exactly who caused an accident, for the same reason that deciding who’s liable in the legal system is hard.
It seems quite easy to say that the person who sabotaged the stop sign was at fault for the accident. What if the saboteur poured oil on the road instead? Is it their fault if the car crashes from sliding on the oil? Okay, they’re being malicious, so they’re at fault. But what if the oil spill was an accident from a truck tipping over? Is it the truck driver’s fault? What if the road was slippery because of ice? Nobody causes the weather, right? On the contrary: the city could’ve cleared and salted the roads earlier, but they didn’t. In the counterfactual world where they did it earlier, the accident wouldn’t have happened.
Okay, how about instead of backward chaining forever, we just check whether the system could have avoided the accident in the counterfactual where it took different actions. The problem is: even in the case where an adversarial stop sign leads to the car crashing, the system potentially could’ve avoided it. Stop signs are placed by humans somewhat arbitrarily using heuristics to determine if an intersection is risky. Shouldn’t the system be able to tell that an intersection is risky, even when there truly isn’t a stop sign there?
The paper tackles the problem by formalizing which behaviors and assumptions regarding the movement of cars and pedestrians are “reasonable” or “unreasonable”, then proving within the toy model that only unreasonable behavior leads to crashes. Makes sense, but in the real world people don’t just follow paths, they do all kinds of things that influence the world. Wouldn’t the legal system be simple if we could just use equations like these to determine liability? I’m just not sure we should expect to eventually cover the long tail of potential situations sufficiently enough to make “provably safe” meaningful.
Also, I’m concerned because they don’t seem to describe it as a toy model despite the extremely simplified set of things they’re considering, and there might be questionable incentives at play to make it seem more sound than it is. From another document on their website:
We believe the industry can come together to create a collaborative platform which provides a “safety seal” that first and foremost will create a safer product, but at the same time will protect OEMs from unreasonable and unwarranted liability assignment by regulators and society.
So they want the cars to be safe, but they also want to avoid liability by proving the accident was someone else’s fault.
(Disclaimer: I’m not very knowledgeable about safety engineering or formal proofs)
I notice that whenever someone brings up “What if this unexpected thing happens?”, you emphasize that it’s about not causing accidents. I’m worried that it’s hard to define exactly who caused an accident, for the same reason that deciding who’s liable in the legal system is hard.
It seems quite easy to say that the person who sabotaged the stop sign was at fault for the accident. What if the saboteur poured oil on the road instead? Is it their fault if the car crashes from sliding on the oil? Okay, they’re being malicious, so they’re at fault. But what if the oil spill was an accident from a truck tipping over? Is it the truck driver’s fault? What if the road was slippery because of ice? Nobody causes the weather, right? On the contrary: the city could’ve cleared and salted the roads earlier, but they didn’t. In the counterfactual world where they did it earlier, the accident wouldn’t have happened.
Okay, how about instead of backward chaining forever, we just check whether the system could have avoided the accident in the counterfactual where it took different actions. The problem is: even in the case where an adversarial stop sign leads to the car crashing, the system potentially could’ve avoided it. Stop signs are placed by humans somewhat arbitrarily using heuristics to determine if an intersection is risky. Shouldn’t the system be able to tell that an intersection is risky, even when there truly isn’t a stop sign there?
The paper tackles the problem by formalizing which behaviors and assumptions regarding the movement of cars and pedestrians are “reasonable” or “unreasonable”, then proving within the toy model that only unreasonable behavior leads to crashes. Makes sense, but in the real world people don’t just follow paths, they do all kinds of things that influence the world. Wouldn’t the legal system be simple if we could just use equations like these to determine liability? I’m just not sure we should expect to eventually cover the long tail of potential situations sufficiently enough to make “provably safe” meaningful.
Also, I’m concerned because they don’t seem to describe it as a toy model despite the extremely simplified set of things they’re considering, and there might be questionable incentives at play to make it seem more sound than it is. From another document on their website:
So they want the cars to be safe, but they also want to avoid liability by proving the accident was someone else’s fault.