Yeah, that is kind of tricky. Let me try to explain what Eliezer_Yudkowsky meant in terms of my preferred form of the Bayes Theorem:
O(H|E) = O(H) * P(E|H) / P(E|~H)
where O indicates odds instead of probability and | indicates “given”.
In other words, “any time you observe evidence, amplify the odds you assign to your beliefs by the probability of observing the evidence if the belief were true, divided by the probabily of observing it if the belief were false.”
Also, keep in mind that Eliezer_Yudkowsky has written about how you should treat very low probability events as being “impossible”, even though you have to assign a non-zero probability to everything.
Nevertheless, his statement still isn’t literally true. The strength of the evidence depends on the ratio P(E|H)/P(E|~H), while the quoted statement only refers to the denominator. So there can be situations where you have 100:1 odds of seeing E if the hypothesis were true, but 1:1000 odds (about a 0.1% chance) of seeing E if it were false.
Such evidence is very strong—it forces you to amplify the odds you assign to H by a factor of 100,000 -- but it’s far from evidence you “couldn’t possibly find”, which to me means something like 1:10^-10 odds.
Still, Eliezer_Yudkowsky is right that, generally, strong evidence will have a very small denominator.
Yeah, that is kind of tricky. Let me try to explain what Eliezer_Yudkowsky meant in terms of my preferred form of the Bayes Theorem:
O(H|E) = O(H) * P(E|H) / P(E|~H)
where O indicates odds instead of probability and | indicates “given”.
In other words, “any time you observe evidence, amplify the odds you assign to your beliefs by the probability of observing the evidence if the belief were true, divided by the probabily of observing it if the belief were false.”
Also, keep in mind that Eliezer_Yudkowsky has written about how you should treat very low probability events as being “impossible”, even though you have to assign a non-zero probability to everything.
Nevertheless, his statement still isn’t literally true. The strength of the evidence depends on the ratio P(E|H)/P(E|~H), while the quoted statement only refers to the denominator. So there can be situations where you have 100:1 odds of seeing E if the hypothesis were true, but 1:1000 odds (about a 0.1% chance) of seeing E if it were false.
Such evidence is very strong—it forces you to amplify the odds you assign to H by a factor of 100,000 -- but it’s far from evidence you “couldn’t possibly find”, which to me means something like 1:10^-10 odds.
Still, Eliezer_Yudkowsky is right that, generally, strong evidence will have a very small denominator.
EDIT: added link
In comments like this, we should link to the existing pages of the wiki, or create stubs of the new ones.
Bayes’ theorem on LessWrong wiki.