If I can make my point a bit more carefully: I don’t think this post successfully surfaces the bits of your model that hypothetical Bob doubts. The claim that “historical accidents are a good reference class for existential catastrophe” is the primary claim at issue. If they were a good reference class, very high risk would obviously be justified, in my view.
Given that your post misses this, I don’t think it succeeds as an defence of high P(doom).
I think a defence of high P(doom) that addresses the issue above would be quite valuable.
Also, for what it’s worth, I treat “I’ve gamed this out a lot and it seems likely to me” as very weak evidence except in domains where I have a track record of successful predictions or proving theorems that match my intuitions. Before I have learned to do either of these things, my intuitions are indeed pretty unreliable!
Yeah I don’t think the arguments in this post on its own should convince that P(doom) is high you if you’re skeptical. There’s lots to say here that doesn’t fit into the post, eg an object-level argument for why AI alignment is “default-failure” / “disjunctive”.
If I can make my point a bit more carefully: I don’t think this post successfully surfaces the bits of your model that hypothetical Bob doubts. The claim that “historical accidents are a good reference class for existential catastrophe” is the primary claim at issue. If they were a good reference class, very high risk would obviously be justified, in my view.
Given that your post misses this, I don’t think it succeeds as an defence of high P(doom).
I think a defence of high P(doom) that addresses the issue above would be quite valuable.
Also, for what it’s worth, I treat “I’ve gamed this out a lot and it seems likely to me” as very weak evidence except in domains where I have a track record of successful predictions or proving theorems that match my intuitions. Before I have learned to do either of these things, my intuitions are indeed pretty unreliable!
Yeah I don’t think the arguments in this post on its own should convince that P(doom) is high you if you’re skeptical. There’s lots to say here that doesn’t fit into the post, eg an object-level argument for why AI alignment is “default-failure” / “disjunctive”.