How about the distinction between (A) “An AGI kills every human, and the people who turned on the AGI didn’t want that to happen” versus (B) “An AGI kills every human, and the people who turned on the AGI did want that to happen”?
I’m guessing that you’re going to say “That’s not a useful distinction because (B) is stupid. Obviously nobody is talking about (B)”. In which case, my response is “The things that are obvious to you and me are not necessarily obvious to people who are new to thinking carefully about AGI x-risk.”
…And in particular, normal people sometimes seem to have an extraordinarily strong prior that “when people are talking about x-risk, it must be (B) and not (A), because (A) is weird sci-fi stuff and (B) is a real thing that could happen”, even after the first 25 times that I insist that I’m really truly talking about (A).
So I do think drawing a distinction between (A) and (B) is a very useful thing to be able to do. What terminology would you suggest for that?
How about the distinction between (A) “An AGI kills every human, and the people who turned on the AGI didn’t want that to happen” versus (B) “An AGI kills every human, and the people who turned on the AGI did want that to happen”?
I think the misuse vs. accident dichotomy is clearer when you don’t focus exclusively on “AGI kills every human” risks. (E.g., global totalitarianism risks strike me as small but non-negligible if we solve the alignment problem. Larger are risks that fall short of totalitarianism but still involve non-morally-humble developers damaging humanity’s long-term potential.)
The dichotomy is really just “AGI does sufficiently bad stuff, and the developers intended this” versus “AGI does sufficiently bad stuff, and the developers didn’t intend this”. The terminology might be non-ideal, but the concepts themselves are very natural.
It’s basically the same concept as “conflict disaster” versus “mistake disaster”. If something falls into both category to a significant extent (e.g., someone tries to become dictator but fails to solve alignment), then it goes in the “accident risk” bucket, because it doesn’t actually matter what you wanted to do with the AI if you’re completely unable to achieve that goal. The dynamics and outcome will end up looking basically the same as other accidents.
“Concrete Problems in AI Safety” used this distinction to make this point, and I think it was likely a useful simplification in that context. I generally think spelling it out is better, and I think people will pattern match your concerns onto the “the sci-fi scenario where AI spontaneously becomes conscious, goes rogue, and pursues its own goal” or “boring old robustness problems” if you don’t invoke structural risk. I think structural risk plays a crucial role in the arguments, and even if you think things that look more like pure accidents are more likely, I think the structural risk story is more plausible to more people and a sufficient cause for concern.
How about the distinction between (A) “An AGI kills every human, and the people who turned on the AGI didn’t want that to happen” versus (B) “An AGI kills every human, and the people who turned on the AGI did want that to happen”?
I’m guessing that you’re going to say “That’s not a useful distinction because (B) is stupid. Obviously nobody is talking about (B)”. In which case, my response is “The things that are obvious to you and me are not necessarily obvious to people who are new to thinking carefully about AGI x-risk.”
…And in particular, normal people sometimes seem to have an extraordinarily strong prior that “when people are talking about x-risk, it must be (B) and not (A), because (A) is weird sci-fi stuff and (B) is a real thing that could happen”, even after the first 25 times that I insist that I’m really truly talking about (A).
So I do think drawing a distinction between (A) and (B) is a very useful thing to be able to do. What terminology would you suggest for that?
I think the misuse vs. accident dichotomy is clearer when you don’t focus exclusively on “AGI kills every human” risks. (E.g., global totalitarianism risks strike me as small but non-negligible if we solve the alignment problem. Larger are risks that fall short of totalitarianism but still involve non-morally-humble developers damaging humanity’s long-term potential.)
The dichotomy is really just “AGI does sufficiently bad stuff, and the developers intended this” versus “AGI does sufficiently bad stuff, and the developers didn’t intend this”. The terminology might be non-ideal, but the concepts themselves are very natural.
It’s basically the same concept as “conflict disaster” versus “mistake disaster”. If something falls into both category to a significant extent (e.g., someone tries to become dictator but fails to solve alignment), then it goes in the “accident risk” bucket, because it doesn’t actually matter what you wanted to do with the AI if you’re completely unable to achieve that goal. The dynamics and outcome will end up looking basically the same as other accidents.
By “intend” do you mean that they sought that outcome / selected for it?
Or merely that it was a known or predictable outcome of their behavior?
I think “unintentional” would already probably be a better term in most cases.
“Concrete Problems in AI Safety” used this distinction to make this point, and I think it was likely a useful simplification in that context. I generally think spelling it out is better, and I think people will pattern match your concerns onto the “the sci-fi scenario where AI spontaneously becomes conscious, goes rogue, and pursues its own goal” or “boring old robustness problems” if you don’t invoke structural risk. I think structural risk plays a crucial role in the arguments, and even if you think things that look more like pure accidents are more likely, I think the structural risk story is more plausible to more people and a sufficient cause for concern.
RE (A): A known side-effect is not an accident.