This part is obviously more speculative, but let’s suppose that Bob reasons thus: given that aliens abduct people, there is perhaps an equal chance that they would abduct any human as any other. (Is this a reasonable assumption? Who knows?) And let’s say that they abduct 100 people per year. So the chance of having been abducted at least once in (let’s say) 40 years of life would be 1 − (7,999,999,900 / 8,000,000,000)^40 = ~0.0000005.
Ah, but there’s a catch! The chance of having an abduction experience if there are aliens isn’t just the chance of being abducted, it’s the chance of being abducted plus the chance of falsely coming to believe you’ve been abducted when in fact you have not. (As surely we do not think that the existence of aliens would prevent humans from having schizophrenic episodes or LSD trips etc.?) Thus we must add, to P(B|A), that 0.001 chance of having a false abduction experience, for a total of 0.0010005. (If we didn’t account for this, we’d end up concluding that Bob’s experience should lead him to revise P(A) drastically down!)
1.000495%. Not 33%. (The prior, we must recall, was 1%.) In other words, this is an update so tiny as to be insignificant.
Of course we can tweak that by modifying our assumptions about the aliens’ behavior—how often do they abduct people, how do they select abductees—but you’d have to start with some truly implausible assumptions to get the numbers anywhere near a large update.
Bob should notice that it is overwhelmingly more likely that his experience was false than that it was real, and it should have essentially no effect whatsoever on his estimate of the probability of the existence of aliens who sometimes abduct people.
Thank you for checking my math and setup! This is my first time trying Bayes in front of an audience.
Yeah, I think I described P(A|B) when trying to describe the sensitivity, you are right that whether aliens actually abduct people given Bob experienced aliens abducting him is P(A|B). It’s possible I need to retract the whole section and example.
Your description of P(B|A) confuses me though. If I think through the standard Bayes mammogram problem, I don’t set P(B|A) as P(“A specific woman gets a positive test result, given some people get a positive test”) and have to figure out what the selection procedure is that the doctor uses to choose people to test. We’re looking for P(“A specific woman gets a positive test result, given she actually has cancer.”) I think Bob gets to start knowing he experienced getting abducted, the same way the woman in the mammogram problem gets to start knowing she got a positive test. He then tries to figure out whether the abduction was aliens or some kind of hallucination, the same way the woman (or her doctor) in the mammogram problem tries to figure out whether the test result is a true positive or a false positive.
Hrm. So, in the mammogram problem, if the sometimes the machine malfunctions in a way that gives a positive result whether or not the woman actually had cancer, then some of the time the woman will coincidentally happen to have cancer when the machine malfunctioned. I think that’s just supposed to be counted as part of the probability the woman with cancer gets a positive test, i.e. the sensitivity? Translating back to Bob’s circumstances, aliens are real, but Bob hallucinated?
Intuitively it makes sense to me that if someone thinks they got abducted by aliens, it’s more likely they’re hallucinating than that they actually got abducted by aliens. It’s true that aliens actually abducting people wouldn’t mean people stop having hallucinations. But adding P(B|¬A) - the rate of false positives—to P(B|A) - the rate of true positives—seems like some kind of weird double counting. What am I misunderstanding here?
Intuitively it makes sense to me that if someone thinks they got abducted by aliens, it’s more likely they’re hallucinating than that they actually got abducted by aliens. It’s true that aliens actually abducting people wouldn’t mean people stop having hallucinations. But adding P(B|¬A) - the rate of false positives—to P(B|A) - the rate of true positives—seems like some kind of weird double counting. What am I misunderstanding here?
Well, to start with, if you don’t include the false positive rate in P(B|A), and work through the numbers, then, as I said, you’ll find that having the abduction experience will drastically lower your probability estimate of aliens. You would have:
So that’s clearly very wrong—even if it doesn’t tell you what the right answer should be.
But, intuitively… well, I explained it already: if aliens exist and abduct people, some people will still take drugs or go crazy or whatever, and hallucinate being abducted. Bob could be one of those people. It’s not double-counting because the A is “aliens exist and abduct people”, not “Bob was abducted by aliens”. (Otherwise P(A) could not possibly have started as high as 0.01—that would’ve been wrong by many orders of magnitude as a prior!) (This is essentially @clone of saturn’s explanation, so see his sibling comment for more on this point.)
Yeah, I think I described P(A|B) when trying to describe the sensitivity, you are right that whether aliens actually abduct people given Bob experienced aliens abducting him is P(A|B). It’s possible I need to retract the whole section and example.
I agree. But I don’t think that you should discard the text entirely, because it seems to me that there is actually a lesson here.
I have had this experience many times: someone (sometimes on this very website) will say something like, “I know for a fact that X; my experience proves it to me beyond any doubt; I accept that my account of it won’t convince you of X, but I at least am certain of it”.
And what I often think in such cases (but perhaps too rarely say) is:
“But you shouldn’t be certain of it. It’s not just that I don’t believe X, merely based on your experience. It’s that you shouldn’t believe X, merely based on your experience. You, yourself, have not seen nearly enough evidence to convince you of X—if you were being a proper Bayesian about it. Not just my, but your conclusion, should be that, actually, X is probably false. Your experience is insufficient to convince me, but it should not have convinced you, either!”
(This is related to something that Robyn Dawes talks about in Rational Choice in an Uncertain World, when he says that people are often too eager to learn from experience.)
This is also related to what E. T. Jaynes calls “resurrection of dead hypotheses”. If you have an alien abduction experience, then this should indeed raise your probability estimate of aliens existing and abducting people. But it should also raise your probability estimate of you being crazy and having hallucinations (to take one example). And since the latter was much more probable than the former to begin with, and the evidence was compatible with both possibilities, observing the evidence cannot result in our coming to believe the former rather than the latter. As Jaynes says (in reference to his example of whether evidence of psychic powers should make one believe in psychic powers):
…Indeed, the very evidence which the ESPers throw at us to convince us, has the opposite effect on our state of belief; issuing reports of sensational data defeats its own purpose. For if the prior probability of deception is greater than that of ESP, then the more improbable the alleged data are on the null hypothesis of no deception and no ESP, the more strongly we are led to believe, not in ESP, but in deception. For this reason, the advocates of ESP (or any other marvel) will never succeed in persuading scientists that their phenomenon is real, until they learn how to eliminate the possibility of deception in the mind of the reader.
The mammogram problem is different because you’re only trying to determine whether a specific woman has cancer, not whether cancer exists at all as a phenomenon. If Bob was abducted by aliens, it implies that alien abduction is real, but the converse isn’t true. You either need to do two separate Bayesian updates (what’s the probability that Bob was abducted given his experience, and then what’s the probability of aliens given the new probability that Bob was abducted), or you need a joint distribution covering all possibilities (Bob not abducted, aliens not real; Bob not abducted, aliens real; Bob abducted, aliens real).
Hrm. Maybe the slip is accidentally switching whether I’m looking for “do aliens abduct people, given Bob experienced being abducted” vs “was Bob’s abduction real, given Bob experienced being abducted.”
But if Bob’s abduction was real, then aliens do abduct people. It would still count even if his was the only actual abduction in the history of the human race. Seems like this isn’t the source of the math not working?
This part is obviously more speculative, but let’s suppose that Bob reasons thus: given that aliens abduct people, there is perhaps an equal chance that they would abduct any human as any other. (Is this a reasonable assumption? Who knows?) And let’s say that they abduct 100 people per year. So the chance of having been abducted at least once in (let’s say) 40 years of life would be 1 − (7,999,999,900 / 8,000,000,000)^40 = ~0.0000005.
Ah, but there’s a catch! The chance of having an abduction experience if there are aliens isn’t just the chance of being abducted, it’s the chance of being abducted plus the chance of falsely coming to believe you’ve been abducted when in fact you have not. (As surely we do not think that the existence of aliens would prevent humans from having schizophrenic episodes or LSD trips etc.?) Thus we must add, to P(B|A), that 0.001 chance of having a false abduction experience, for a total of 0.0010005. (If we didn’t account for this, we’d end up concluding that Bob’s experience should lead him to revise P(A) drastically down!)
So, the revised calculation:
P(A|B) = (0.0010005 * 0.01) / ((0.0010005 * 0.01) + (0.001 * 0.99)) = 0.000010005 / (0.000010005 + 0.00099) = 0.000010005 / 0.001000005 = ~0.01000495 = 1.000495%.
1.000495%. Not 33%. (The prior, we must recall, was 1%.) In other words, this is an update so tiny as to be insignificant.
Of course we can tweak that by modifying our assumptions about the aliens’ behavior—how often do they abduct people, how do they select abductees—but you’d have to start with some truly implausible assumptions to get the numbers anywhere near a large update.
Bob should notice that it is overwhelmingly more likely that his experience was false than that it was real, and it should have essentially no effect whatsoever on his estimate of the probability of the existence of aliens who sometimes abduct people.
Thank you for checking my math and setup! This is my first time trying Bayes in front of an audience.
Yeah, I think I described P(A|B) when trying to describe the sensitivity, you are right that whether aliens actually abduct people given Bob experienced aliens abducting him is P(A|B). It’s possible I need to retract the whole section and example.
Your description of P(B|A) confuses me though. If I think through the standard Bayes mammogram problem, I don’t set P(B|A) as P(“A specific woman gets a positive test result, given some people get a positive test”) and have to figure out what the selection procedure is that the doctor uses to choose people to test. We’re looking for P(“A specific woman gets a positive test result, given she actually has cancer.”) I think Bob gets to start knowing he experienced getting abducted, the same way the woman in the mammogram problem gets to start knowing she got a positive test. He then tries to figure out whether the abduction was aliens or some kind of hallucination, the same way the woman (or her doctor) in the mammogram problem tries to figure out whether the test result is a true positive or a false positive.
Hrm. So, in the mammogram problem, if the sometimes the machine malfunctions in a way that gives a positive result whether or not the woman actually had cancer, then some of the time the woman will coincidentally happen to have cancer when the machine malfunctioned. I think that’s just supposed to be counted as part of the probability the woman with cancer gets a positive test, i.e. the sensitivity? Translating back to Bob’s circumstances, aliens are real, but Bob hallucinated?
Intuitively it makes sense to me that if someone thinks they got abducted by aliens, it’s more likely they’re hallucinating than that they actually got abducted by aliens. It’s true that aliens actually abducting people wouldn’t mean people stop having hallucinations. But adding P(B|¬A) - the rate of false positives—to P(B|A) - the rate of true positives—seems like some kind of weird double counting. What am I misunderstanding here?
Well, to start with, if you don’t include the false positive rate in P(B|A), and work through the numbers, then, as I said, you’ll find that having the abduction experience will drastically lower your probability estimate of aliens. You would have:
P(A|B) = (0.000005 · 0.01) / ((0.000005 · 0.01) + (0.001 · 0.99)) = 0.00000005 / (0.00000005 + 0.00099) = 0.00000005 / 0.00099005 = ~0.0000505025 = 0.00505025%.
So that’s clearly very wrong—even if it doesn’t tell you what the right answer should be.
But, intuitively… well, I explained it already: if aliens exist and abduct people, some people will still take drugs or go crazy or whatever, and hallucinate being abducted. Bob could be one of those people. It’s not double-counting because the A is “aliens exist and abduct people”, not “Bob was abducted by aliens”. (Otherwise P(A) could not possibly have started as high as 0.01—that would’ve been wrong by many orders of magnitude as a prior!) (This is essentially @clone of saturn’s explanation, so see his sibling comment for more on this point.)
I agree. But I don’t think that you should discard the text entirely, because it seems to me that there is actually a lesson here.
I have had this experience many times: someone (sometimes on this very website) will say something like, “I know for a fact that X; my experience proves it to me beyond any doubt; I accept that my account of it won’t convince you of X, but I at least am certain of it”.
And what I often think in such cases (but perhaps too rarely say) is:
“But you shouldn’t be certain of it. It’s not just that I don’t believe X, merely based on your experience. It’s that you shouldn’t believe X, merely based on your experience. You, yourself, have not seen nearly enough evidence to convince you of X—if you were being a proper Bayesian about it. Not just my, but your conclusion, should be that, actually, X is probably false. Your experience is insufficient to convince me, but it should not have convinced you, either!”
(EDIT: For example.)
(This is related to something that Robyn Dawes talks about in Rational Choice in an Uncertain World, when he says that people are often too eager to learn from experience.)
This is also related to what E. T. Jaynes calls “resurrection of dead hypotheses”. If you have an alien abduction experience, then this should indeed raise your probability estimate of aliens existing and abducting people. But it should also raise your probability estimate of you being crazy and having hallucinations (to take one example). And since the latter was much more probable than the former to begin with, and the evidence was compatible with both possibilities, observing the evidence cannot result in our coming to believe the former rather than the latter. As Jaynes says (in reference to his example of whether evidence of psychic powers should make one believe in psychic powers):
The mammogram problem is different because you’re only trying to determine whether a specific woman has cancer, not whether cancer exists at all as a phenomenon. If Bob was abducted by aliens, it implies that alien abduction is real, but the converse isn’t true. You either need to do two separate Bayesian updates (what’s the probability that Bob was abducted given his experience, and then what’s the probability of aliens given the new probability that Bob was abducted), or you need a joint distribution covering all possibilities (Bob not abducted, aliens not real; Bob not abducted, aliens real; Bob abducted, aliens real).
Hrm. Maybe the slip is accidentally switching whether I’m looking for “do aliens abduct people, given Bob experienced being abducted” vs “was Bob’s abduction real, given Bob experienced being abducted.”
But if Bob’s abduction was real, then aliens do abduct people. It would still count even if his was the only actual abduction in the history of the human race. Seems like this isn’t the source of the math not working?