I want to note that at least for me I’m mostly arguing about the strength of the update. Like, for a claim of the form “An accident with powerful technology X will directly cause human extinction”, knowing nothing else I’d probably start with some pretty low prior (it’s quite concrete, naming a specific technology X, which upper bounds the average probability across Xs at 1/X, and humans tend to try not to go extinct). Let’s say 0.1%, which feels high for an average X, but maybe AI is really powerful even among powerful technologies.
From that point you update on 1) arguments for technical AI risk and 2) arguments for failure of humanity’s response. In my case (the “Them” side), this gets to ~10%, which in log-odds (base 10) is an update of about +2. In contrast, if Buck’s position were 50% risk, that would be an update of about +3, and at 90% risk, an update of about +4.
I’m not claiming that I started with some prior and decided on some evidence likelihood and computed the update—I didn’t—I more want to illustrate that we all agree on the sign of the initial update, and agree that the update should be strong, so I think when arguing against the “Them” position it’s important to note that you’re arguing for an even stronger update, which requires a correspondingly higher evidence likelihood. When people give me single examples of things going wrong, I feel tempted to say “where do you think the 10% comes from?”
I want to note that at least for me I’m mostly arguing about the strength of the update. Like, for a claim of the form “An accident with powerful technology X will directly cause human extinction”, knowing nothing else I’d probably start with some pretty low prior (it’s quite concrete, naming a specific technology X, which upper bounds the average probability across Xs at 1/X, and humans tend to try not to go extinct). Let’s say 0.1%, which feels high for an average X, but maybe AI is really powerful even among powerful technologies.
From that point you update on 1) arguments for technical AI risk and 2) arguments for failure of humanity’s response. In my case (the “Them” side), this gets to ~10%, which in log-odds (base 10) is an update of about +2. In contrast, if Buck’s position were 50% risk, that would be an update of about +3, and at 90% risk, an update of about +4.
I’m not claiming that I started with some prior and decided on some evidence likelihood and computed the update—I didn’t—I more want to illustrate that we all agree on the sign of the initial update, and agree that the update should be strong, so I think when arguing against the “Them” position it’s important to note that you’re arguing for an even stronger update, which requires a correspondingly higher evidence likelihood. When people give me single examples of things going wrong, I feel tempted to say “where do you think the 10% comes from?”