I’m going to try another explanation that I hope isn’t too redundant with Benja’s.
Consider the events
W = The LHC would destroy Earth
F = the LHC fails to operate
S = we survive (= F OR not W)
We want to know P(W|F) or P(W|F,S), so let’s apply Bayes.
First thing to note is that since F ⇒ S, we have P(W|F) = P(W|F,S), so we can just work out P(W|F)
Bayes:
P(W|F) = P(F|W)P(W)/P(F)
Note that none of these probabilities are conditional on survival. So unless in the absence of any selection effects the probability of failure still depends on whether the LHC would destroy Earth, P(F|W) = P(F), and thus P(W|F) = P(W).
(I suppose one could argue that a failure could be caused by a new law of physics that would also lead the LHC to destroy the Earth, but that isn’t what is being argued here—at least so I think; my apologies to anyone who is arguing that)
In effect what Eliezer and many commenters are doing is substituting P(F|W,S) for P(F|W). These probabilities are not the same and so this substitution is illegitimate.
Benja, I also think of it that way intuitively. I would like to add though that it doesn’t really matter whether you have branches or just a single nondeterministic world—Bayes’ theorem applies the same either way.
I’m going to try another explanation that I hope isn’t too redundant with Benja’s.
Consider the events
W = The LHC would destroy Earth F = the LHC fails to operate S = we survive (= F OR not W)
We want to know P(W|F) or P(W|F,S), so let’s apply Bayes.
First thing to note is that since F ⇒ S, we have P(W|F) = P(W|F,S), so we can just work out P(W|F)
Bayes:
P(W|F) = P(F|W)P(W)/P(F)
Note that none of these probabilities are conditional on survival. So unless in the absence of any selection effects the probability of failure still depends on whether the LHC would destroy Earth, P(F|W) = P(F), and thus P(W|F) = P(W).
(I suppose one could argue that a failure could be caused by a new law of physics that would also lead the LHC to destroy the Earth, but that isn’t what is being argued here—at least so I think; my apologies to anyone who is arguing that)
In effect what Eliezer and many commenters are doing is substituting P(F|W,S) for P(F|W). These probabilities are not the same and so this substitution is illegitimate.
Benja, I also think of it that way intuitively. I would like to add though that it doesn’t really matter whether you have branches or just a single nondeterministic world—Bayes’ theorem applies the same either way.