Seems to me that this is building in too much content / will have the wrong connotations. If an ML researcher hears about “recklessness risk”, they’re not unlikely to go “oh, well I don’t feel ‘reckless’ at my day job, so I’m off the hook”.
Locating the issue in the cognition of the developer is probably helpful in some contexts, but it has the disadvantage that (a) people will reflect on their cognition, not notice “negligent-feeling thoughts”, and conclude that accident risk is low; and (b) it encourages people to take the eye off the ball, focusing on psychology (and arguments about whose psychology is X versus Y) rather than focusing on properties of the AI itself.
“Accident risk” is maybe better just because it’s vaguer. The main problem I see with it isn’t “this sounds like it’s letting the developers off the hook” (since when do we assume that all accidents are faultless?). Rather, I think the problem with “accident” is that it sounds minor.
Accidentally breaking a plate is an “accident”. Accidentally destroying a universe is… something a bit worse than that.
Instead of “accident”, we could say “gross negligence” or “recklessness” for catastrophic risk from AI misalignment.
Seems to me that this is building in too much content / will have the wrong connotations. If an ML researcher hears about “recklessness risk”, they’re not unlikely to go “oh, well I don’t feel ‘reckless’ at my day job, so I’m off the hook”.
Locating the issue in the cognition of the developer is probably helpful in some contexts, but it has the disadvantage that (a) people will reflect on their cognition, not notice “negligent-feeling thoughts”, and conclude that accident risk is low; and (b) it encourages people to take the eye off the ball, focusing on psychology (and arguments about whose psychology is X versus Y) rather than focusing on properties of the AI itself.
“Accident risk” is maybe better just because it’s vaguer. The main problem I see with it isn’t “this sounds like it’s letting the developers off the hook” (since when do we assume that all accidents are faultless?). Rather, I think the problem with “accident” is that it sounds minor.
Accidentally breaking a plate is an “accident”. Accidentally destroying a universe is… something a bit worse than that.
Fair point.
If the issue with “accident” is that it sounds minor*, then one could say “catastrophic accident risk” or similar.
*I’m not fully bought into this as the main issue, but supposing that it is...