Have you reviewed the paper? (It is the first link under “The RSS Concept” in the page which was linked to before, though perhaps I should have linked to it directly.) It seems to lay out the proof, and discusses pedestrians, and deals with most of the objections you’re raising, including obstructions and driving off of marked roads. I admit I have not worked through the proof in detail, but I have read through it, and my understanding is that it was accepted, and a large literature has been built that extends it.
And the objections about slippery roads and braking are the set of things I noted under “traditional engineering analysis and failure rates” I agree that guarantees are non-trivial, but they also aren’t outside of what is done already in safety analysis, and there is explicit work in the literature on the issue, both from the verification and validation side, and from the perception and sensing weather conditions side.
I looked at a bunch of the articles and rules listed on the website but didn’t review the paper in-depth. The rules don’t mention pedestrians, but you are right that the paper does has an extremely cursory treatment of them. It basically says “assume the pedestrians are also following RSS”, which of course, is in direct contradiction to your statement that if all cars on the road follow RSS, that there will be no accidents, since pedestrians are of course part of the road environment, and they do not follow RSS in-reality (and as such RSS cannot prove that no collisions with pedestrians will occur, or that the only way to do a reasonable thing when interfacing with pedestrians will require some collision with another vehicle).
Look, I think the paper makes some valid proofs, but it is completely (and IMO obviously) inaccurate to say that RSS proves that if all cars follow RSS, that no accidents will take place.
What it of course proves, as all formal system analysis does, is that in an extremely simplified toy world, that no such accidents occur. The applicability of that toy world to the analysis of real situations is then of course a classical problem that engineering professionals face every day. But that transfer is the whole central issue at stake in arguing for “provably safe AI” and so this article is absolutely not the place to just glance over the extremely difficult and crucial step of translating a form toy world like RSS into real world outcomes, and in this case you straightforwardly made an inaccurate unqualified statement.
To start at the end, you claim I “straightforwardly made an inaccurate unqualified statement,” but replaced my statement about “what a car needs to do not to cause accidents” with “no accidents will take place.” And I certainly agree that there is an “extremely difficult and crucial step of translating a form toy world like RSS into real world outcomes,” but the toy model that the paper is dealing with is therefore one of rule-following entities, both pedestrians and cars. That’s why it’s not going to require accounting for “what if pedestrians do something illegal and unexpected.”
Of course, I agree that this drastically limits the proof, or as I said initially, “relying on assumptions about other car behavior is a limit to provable safety,” but you seem to insist that because the proof doesn’t do something I never claimed it did, it’s glossing over something.
That said, I agree that I did not discuss pedestrians, but as you sort-of admit, the paper does—it treats stationary pedestrians not at crosswalks, and not on sidewalks, as largely unpredictable entities that may enter the road. For example, it notes that “even if pedestrians do not have priority, if they entered the road at a safe distance, cars must brake and let them pass.” But again, you’re glossing over the critical assumption for the entire section, which is responsibility for accidents. And this is particularly critical; the claim is not that pedestrians and other cars cannot cause accidents, but that the safe car will not do so.
Given all of that, to get back to the beginning, your initial position was that “RSS seems miles away from anything that one could describe as a formalization of how to avoid an accident.” Do you agree that it’s close to “a formalization of how to avoid causing an accident”?
To start at the end, you claim I “straightforwardly made an inaccurate unqualified statement,” but replaced my statement about “what a car needs to do not to cause accidents” with “no accidents will take place.”
I don’t think I msisquoted you. You said:
In the case of RSS, it’s provable that if other cars follow the law, and/or all cars on the road abide by the rules, those cars will only have accidents if their sensors are incorrect.
You did not say “cause” you said “have”. I did read your words carefully. You said all accidents would be the result of sensor-defects. This means that if there are no sensor-defects, no accidents will take place. That’s a straightforwardly inaccurate statement about what RSS proves.
I don’t think either of us inherently cares about “responsibility”. I think the theme of that section (as well as other sections) was pretty clearly that we could formally prove certain outcomes impossible.
“Causing” is a confusing word. I don’t think RSS proves at all that a car could not cause an accident. I haven’t engaged with the formalism in-detail, but as far as I can tell it would totally be compatible with a car driving extremely recklessly in a pedestrian environment due to making assumptions about pedestrian behavior that are not accurate. If you really care, I could formally give you an example given its modeling assumptions about pedestrians, but I am pretty sure you could do the same.
Yes, after saying it was about what they need “to do not to cause accidents” and that “any accidents which could occur will be attributable to other cars actions,” which I then added caveats to regarding pedestrians, I said “will only have accidents” when I should have said “will only cause accidents.” I have fixed that with another edit. But I think you’re confused about what I’m trying to show .
Principally, I think you are wrong about what needs to be shown here for safety in the sense I outlined, or are trying to say that the sense I outlined doesn’t lead to something I don’t claim. If what is needed in order for a system to be safe is that no damage will be caused in situations which involve the system, you’re heading in the direction of a claim that the only safe AI is a universal dictator that has sufficient power to control all outcomes. My claim, on the other hand, is that in sociotechnological systems, the way that safety is achieved is by creating guarantees that each actor—human or AI—behaves according to rules that minimizes foreseeable dangers. That would include safeguards for stupid, malicious, or dangerous human actions, much like human systems have laws about dangerous actions. However, in a domain like driving, in the same way that it’s impossible for human drivers to both get where they are going, and never hit pedestrians who act erratically and jump out from behind obstacles into the road with an oncoming car, a safe autonomous vehicle wouldn’t be expected to solve every possible case of human misbehavior—just to drive responsibly.
More specifically, you make the claim that “as far as I can tell it would totally be compatible with a car driving extremely recklessly in a pedestrian environment due to making assumptions about pedestrian behavior that are not accurate.” The paper, on the other hand, says “For example, in a typical residential street, a pedestrian has the priority over the vehicles, and it follows that vehicles must yield and be cautious with respect to pedestrians,” and formalizes this with statements like “a vehicle must be in a kinematic state such that if it will apply a proper response (acceleration for ρ seconds and when braking) it will remain outside of a ball of radius 50cm around the pedestrian.”
I also think that it formalizes reasonable behavior for pedestrians, but I agree that it won’t cover every case—pedestrians oblivious to cars that are driving in ways that are otherwise safe, who rapidly change their path to jump in front of cars, are sometimes able to be hit by those cars—but I think fault is pretty clear here. (And the paper is clear that even in those cases, the car would need to both drive safely in residential areas, and attempt to brake or avoid the pedestrian in order to avoid crashes even in cases with irresponsible and erratic humans!)
But again, as I said initially, this isn’t solving the general case of AI safety, it’s solving a much narrower problem. And if you wanted to make the case that this isn’t enough for similar scenarios that we care about, I will strongly agree that for more capable systems, the set of situations it would need to avoid are correspondingly larger, and the set of necessary guarantees are far stronger. But as I said at the beginning, I’m not making that argument—just the much simpler one that proveability can work in physical systems, and can be applied in sociotechnological systems in ways that make sense.
The crux of these types of arguments seems to be conflating the provable safety of an agent in a system with the expectation of absolute safety. In my experience, this is the norm, not the exception, and needs to be explicitly addressed.
In agreement with what you posted above, I think it is formally trivial to construct a scenario in which a pedestrian jumps in front of a car, making it provably impossible for a vehicle to stop in time to avoid a collision using high school physics.
Likewise, I have the intuition that AI safety, in general, should have various “no-go theorems” about unprovability outside a reasonable problem scope or that finding such proofs would be np-hard or worse. If you know of any specific results( outside of general computability theory) , could you please share them? It would be nice if the community could avoid falling into the trap of trying to prove too much.
(Sorry if this isn’t the correct location for this post.)
On the absolute safety, I very much like the way you put it, and will likely use that framing in the future, so thanks!
On impossibility results, there are some, andI definitely think that this is a good question, but also agree this isn’t quite the right place to ask. I’d suggest talking to some of the agents foundations people for suggestions
Have you reviewed the paper? (It is the first link under “The RSS Concept” in the page which was linked to before, though perhaps I should have linked to it directly.) It seems to lay out the proof, and discusses pedestrians, and deals with most of the objections you’re raising, including obstructions and driving off of marked roads. I admit I have not worked through the proof in detail, but I have read through it, and my understanding is that it was accepted, and a large literature has been built that extends it.
And the objections about slippery roads and braking are the set of things I noted under “traditional engineering analysis and failure rates” I agree that guarantees are non-trivial, but they also aren’t outside of what is done already in safety analysis, and there is explicit work in the literature on the issue, both from the verification and validation side, and from the perception and sensing weather conditions side.
I looked at a bunch of the articles and rules listed on the website but didn’t review the paper in-depth. The rules don’t mention pedestrians, but you are right that the paper does has an extremely cursory treatment of them. It basically says “assume the pedestrians are also following RSS”, which of course, is in direct contradiction to your statement that if all cars on the road follow RSS, that there will be no accidents, since pedestrians are of course part of the road environment, and they do not follow RSS in-reality (and as such RSS cannot prove that no collisions with pedestrians will occur, or that the only way to do a reasonable thing when interfacing with pedestrians will require some collision with another vehicle).
Look, I think the paper makes some valid proofs, but it is completely (and IMO obviously) inaccurate to say that RSS proves that if all cars follow RSS, that no accidents will take place.
What it of course proves, as all formal system analysis does, is that in an extremely simplified toy world, that no such accidents occur. The applicability of that toy world to the analysis of real situations is then of course a classical problem that engineering professionals face every day. But that transfer is the whole central issue at stake in arguing for “provably safe AI” and so this article is absolutely not the place to just glance over the extremely difficult and crucial step of translating a form toy world like RSS into real world outcomes, and in this case you straightforwardly made an inaccurate unqualified statement.
To start at the end, you claim I “straightforwardly made an inaccurate unqualified statement,” but replaced my statement about “what a car needs to do not to cause accidents” with “no accidents will take place.” And I certainly agree that there is an “extremely difficult and crucial step of translating a form toy world like RSS into real world outcomes,” but the toy model that the paper is dealing with is therefore one of rule-following entities, both pedestrians and cars. That’s why it’s not going to require accounting for “what if pedestrians do something illegal and unexpected.”
Of course, I agree that this drastically limits the proof, or as I said initially, “relying on assumptions about other car behavior is a limit to provable safety,” but you seem to insist that because the proof doesn’t do something I never claimed it did, it’s glossing over something.
That said, I agree that I did not discuss pedestrians, but as you sort-of admit, the paper does—it treats stationary pedestrians not at crosswalks, and not on sidewalks, as largely unpredictable entities that may enter the road. For example, it notes that “even if pedestrians do not have priority, if they entered the road at a safe distance, cars must brake and let them pass.” But again, you’re glossing over the critical assumption for the entire section, which is responsibility for accidents. And this is particularly critical; the claim is not that pedestrians and other cars cannot cause accidents, but that the safe car will not do so.
Given all of that, to get back to the beginning, your initial position was that “RSS seems miles away from anything that one could describe as a formalization of how to avoid an accident.” Do you agree that it’s close to “a formalization of how to avoid causing an accident”?
I don’t think I msisquoted you. You said:
You did not say “cause” you said “have”. I did read your words carefully. You said all accidents would be the result of sensor-defects. This means that if there are no sensor-defects, no accidents will take place. That’s a straightforwardly inaccurate statement about what RSS proves.
I don’t think either of us inherently cares about “responsibility”. I think the theme of that section (as well as other sections) was pretty clearly that we could formally prove certain outcomes impossible.
“Causing” is a confusing word. I don’t think RSS proves at all that a car could not cause an accident. I haven’t engaged with the formalism in-detail, but as far as I can tell it would totally be compatible with a car driving extremely recklessly in a pedestrian environment due to making assumptions about pedestrian behavior that are not accurate. If you really care, I could formally give you an example given its modeling assumptions about pedestrians, but I am pretty sure you could do the same.
Yes, after saying it was about what they need “to do not to cause accidents” and that “any accidents which could occur will be attributable to other cars actions,” which I then added caveats to regarding pedestrians, I said “will only have accidents” when I should have said “will only cause accidents.” I have fixed that with another edit. But I think you’re confused about what I’m trying to show .
Principally, I think you are wrong about what needs to be shown here for safety in the sense I outlined, or are trying to say that the sense I outlined doesn’t lead to something I don’t claim. If what is needed in order for a system to be safe is that no damage will be caused in situations which involve the system, you’re heading in the direction of a claim that the only safe AI is a universal dictator that has sufficient power to control all outcomes. My claim, on the other hand, is that in sociotechnological systems, the way that safety is achieved is by creating guarantees that each actor—human or AI—behaves according to rules that minimizes foreseeable dangers. That would include safeguards for stupid, malicious, or dangerous human actions, much like human systems have laws about dangerous actions. However, in a domain like driving, in the same way that it’s impossible for human drivers to both get where they are going, and never hit pedestrians who act erratically and jump out from behind obstacles into the road with an oncoming car, a safe autonomous vehicle wouldn’t be expected to solve every possible case of human misbehavior—just to drive responsibly.
More specifically, you make the claim that “as far as I can tell it would totally be compatible with a car driving extremely recklessly in a pedestrian environment due to making assumptions about pedestrian behavior that are not accurate.” The paper, on the other hand, says “For example, in a typical residential street, a pedestrian has the priority over the vehicles, and it follows that vehicles must yield and be cautious with respect to pedestrians,” and formalizes this with statements like “a vehicle must be in a kinematic state such that if it will apply a proper response (acceleration for ρ seconds and when braking) it will remain outside of a ball of radius 50cm around the pedestrian.”
I also think that it formalizes reasonable behavior for pedestrians, but I agree that it won’t cover every case—pedestrians oblivious to cars that are driving in ways that are otherwise safe, who rapidly change their path to jump in front of cars, are sometimes able to be hit by those cars—but I think fault is pretty clear here. (And the paper is clear that even in those cases, the car would need to both drive safely in residential areas, and attempt to brake or avoid the pedestrian in order to avoid crashes even in cases with irresponsible and erratic humans!)
But again, as I said initially, this isn’t solving the general case of AI safety, it’s solving a much narrower problem. And if you wanted to make the case that this isn’t enough for similar scenarios that we care about, I will strongly agree that for more capable systems, the set of situations it would need to avoid are correspondingly larger, and the set of necessary guarantees are far stronger. But as I said at the beginning, I’m not making that argument—just the much simpler one that proveability can work in physical systems, and can be applied in sociotechnological systems in ways that make sense.
The crux of these types of arguments seems to be conflating the provable safety of an agent in a system with the expectation of absolute safety. In my experience, this is the norm, not the exception, and needs to be explicitly addressed.
In agreement with what you posted above, I think it is formally trivial to construct a scenario in which a pedestrian jumps in front of a car, making it provably impossible for a vehicle to stop in time to avoid a collision using high school physics.
Likewise, I have the intuition that AI safety, in general, should have various “no-go theorems” about unprovability outside a reasonable problem scope or that finding such proofs would be np-hard or worse. If you know of any specific results( outside of general computability theory) , could you please share them? It would be nice if the community could avoid falling into the trap of trying to prove too much.
(Sorry if this isn’t the correct location for this post.)
On the absolute safety, I very much like the way you put it, and will likely use that framing in the future, so thanks!
On impossibility results, there are some, andI definitely think that this is a good question, but also agree this isn’t quite the right place to ask. I’d suggest talking to some of the agents foundations people for suggestions