I’ve been intermittently discussing Bayes’ Theorem with the uninitiated for years, with uneven results. Typically, I’ll give the classic problem:
3,000 people in the US have Sudden Death Syndrome. I have a test that is 99% accurate; that is, it will wrong on any given person one percent of the time. Steve tests positive for SDS. What is the chance that he has it?
Afterwards, I explain the answer by comparing the false positives to the true positives. And, then I see the Bayes’ Theorem Look, which conveys to me this: “I know Mayne’s good with numbers, and I’m not, so I suppose he’s probably right. Still, this whole thing is some sort of impractical number magic.” Then they nod politely and change the subject, and I save the use of Bayes’ Theorem as a means of solving disagreements for another day.
So this leads to my giving a very short presentation on the Prosecutor’s Fallacy next week. The basics of the fallacy are if you’ve got a one-in-3 million DNA match on a suspect, that doesn’t mean it’s three million-to-one that you’ve got that dude’s DNA. I need to present it to bright, interested people who will go straight to brain freeze if I display any equations at all. This isn’t frequentists-vs.-Bayesians; this is just a simple application of Bayes’ Theorem. (I suspect this will be easier to understand than the medical problem.)
I’ve read Bayesian explanations, but I’m aiming at people who are actively uninterested in learning math, and if I can get them to understand only the Prosecutor’s Fallacy, I’ll call Win. A larger understanding of the underlying structure would be a bigger win. Anyone done something like this before with success (or failure of either educational or entertainment value?)
For this specific case, you could try asking the analogous question with a higher probability value. E.g. “if you’ve got a one-in-two DNA match on a suspect, does that mean it’s one-in-two that you’ve got that dude’s DNA?”. Maybe you can have some graphic that’s meant to represent a several million people, with half of the folks colored as positive matches. When they say “no, it’s not one-in-two”, you can work your way up to the three million case by showing pictures displaying what the estimated amount of hits would be for a 1 to 3, 1 to 5, 1 to 10, 1 to 100, 1 to 1000 etc. case.
In general, try to use examples that are familiar from everyday life (and thus don’t feel like math). For the Bayes’ theorem introduction, you could try “a man comes to a doctor complaining about a headache. The doctor knows that both the flu and brain cancer can cause headaches. If you knew nothing else about the case, which one would you think was more likely?” Then, after they’ve (hopefully) said that the man is more likely to be suffering of a flu, you can mention that brain cancer is much more likely to cause a headache than a flu is, but because flu is so much more common, their answer was nevertheless the correct one.
Most car accidents occur close to people’s homes, not because it’s more dangerous close to home, but because people spend most of their driving time close to their homes.
Most pedestrians who get hit by cars get hit at crosswalks, not because it’s more dangerous at a crosswalk, but because most people cross at crosswalks.
Most women who get raped get raped by people they know, not because strangers are less dangerous than people they know, but because they spend more time around people they know.
Your numbers have me confused. I’d read the grandparent as implying 300M total population, out of which 3000 have the disease. (This is a hint to clarify the info in the grandparent comment btw—whether I’ve made a dire mistake or not.)
Another point to clarify is that the test’s detection power isn’t necessarily the inverse of its false positive rate. Here I assume “99%” characterizes both.
What I get: 300M times 1% false positive means 3M will test positive. Out of the 3000 who have the disease 30 will test negative, 2970 positive. Out of the total population the number who will test positive is 3M+2970 of whom 2970 in fact have the disease, yielding a conditional probability of .98 in 1000 that Steve has SDS.
I think it’s really important to get the idea of a sliding scale of evidentiary strength across to people. (This is something that has occurred to me from some of my recent attempts to explain the Knox case to people without training in Bayesianism.) One’s level of confidence that something is true varies continuously with the strength of the evidence. It’s like a “score” that you’re keeping, with information you hear about moving the score up and down.
The abstract structure of the prosecutor’s fallacy is misjudging the prior probability. People forget that you start with a handicap—and that handicap may be quite substantial. Thus, if a piece of evidence (like a test result) is worth, say “10 points” toward guilt, hearing about that piece of evidence doesn’t necessarily make the score +10 in favor of guilt; if the handicap was, say, −7, then the score is only +3. If, say, a score of +15 is needed for conviction, the prosecution still has a long way to go.
(By the way, did you see my reply to your comment about psychological evidence?)
People understand counting better than probability. Talk about a group of people and how many of them match. (Starting with smaller numbers might help too.)
Bleg for assistance:
I’ve been intermittently discussing Bayes’ Theorem with the uninitiated for years, with uneven results. Typically, I’ll give the classic problem:
3,000 people in the US have Sudden Death Syndrome. I have a test that is 99% accurate; that is, it will wrong on any given person one percent of the time. Steve tests positive for SDS. What is the chance that he has it?
Afterwards, I explain the answer by comparing the false positives to the true positives. And, then I see the Bayes’ Theorem Look, which conveys to me this: “I know Mayne’s good with numbers, and I’m not, so I suppose he’s probably right. Still, this whole thing is some sort of impractical number magic.” Then they nod politely and change the subject, and I save the use of Bayes’ Theorem as a means of solving disagreements for another day.
So this leads to my giving a very short presentation on the Prosecutor’s Fallacy next week. The basics of the fallacy are if you’ve got a one-in-3 million DNA match on a suspect, that doesn’t mean it’s three million-to-one that you’ve got that dude’s DNA. I need to present it to bright, interested people who will go straight to brain freeze if I display any equations at all. This isn’t frequentists-vs.-Bayesians; this is just a simple application of Bayes’ Theorem. (I suspect this will be easier to understand than the medical problem.)
I’ve read Bayesian explanations, but I’m aiming at people who are actively uninterested in learning math, and if I can get them to understand only the Prosecutor’s Fallacy, I’ll call Win. A larger understanding of the underlying structure would be a bigger win. Anyone done something like this before with success (or failure of either educational or entertainment value?)
For this specific case, you could try asking the analogous question with a higher probability value. E.g. “if you’ve got a one-in-two DNA match on a suspect, does that mean it’s one-in-two that you’ve got that dude’s DNA?”. Maybe you can have some graphic that’s meant to represent a several million people, with half of the folks colored as positive matches. When they say “no, it’s not one-in-two”, you can work your way up to the three million case by showing pictures displaying what the estimated amount of hits would be for a 1 to 3, 1 to 5, 1 to 10, 1 to 100, 1 to 1000 etc. case.
In general, try to use examples that are familiar from everyday life (and thus don’t feel like math). For the Bayes’ theorem introduction, you could try “a man comes to a doctor complaining about a headache. The doctor knows that both the flu and brain cancer can cause headaches. If you knew nothing else about the case, which one would you think was more likely?” Then, after they’ve (hopefully) said that the man is more likely to be suffering of a flu, you can mention that brain cancer is much more likely to cause a headache than a flu is, but because flu is so much more common, their answer was nevertheless the correct one.
Other good examples:
Most car accidents occur close to people’s homes, not because it’s more dangerous close to home, but because people spend most of their driving time close to their homes.
Most pedestrians who get hit by cars get hit at crosswalks, not because it’s more dangerous at a crosswalk, but because most people cross at crosswalks.
Most women who get raped get raped by people they know, not because strangers are less dangerous than people they know, but because they spend more time around people they know.
If you’re using Powerpoint, you might want to make a slide that says something like:
2,999 negatives → 1% test positive → 30 false positives
1 positive → 99% test positive → 1 true positive
So out of 31 positive tests, only 1 person has SDS.
If you’ve got the time, use a little horde of stick figures, entering into a testing machine and with test-positive results getting spit out.
Your numbers have me confused. I’d read the grandparent as implying 300M total population, out of which 3000 have the disease. (This is a hint to clarify the info in the grandparent comment btw—whether I’ve made a dire mistake or not.)
Another point to clarify is that the test’s detection power isn’t necessarily the inverse of its false positive rate. Here I assume “99%” characterizes both.
What I get: 300M times 1% false positive means 3M will test positive. Out of the 3000 who have the disease 30 will test negative, 2970 positive. Out of the total population the number who will test positive is 3M+2970 of whom 2970 in fact have the disease, yielding a conditional probability of .98 in 1000 that Steve has SDS.
I fail at reading. I thought it said “ONE in 3000 people in the US....”
Do it with pictures
I take it you’ve already looked at Eliezer’s “Intuitive Explanation”?
I think it’s really important to get the idea of a sliding scale of evidentiary strength across to people. (This is something that has occurred to me from some of my recent attempts to explain the Knox case to people without training in Bayesianism.) One’s level of confidence that something is true varies continuously with the strength of the evidence. It’s like a “score” that you’re keeping, with information you hear about moving the score up and down.
The abstract structure of the prosecutor’s fallacy is misjudging the prior probability. People forget that you start with a handicap—and that handicap may be quite substantial. Thus, if a piece of evidence (like a test result) is worth, say “10 points” toward guilt, hearing about that piece of evidence doesn’t necessarily make the score +10 in favor of guilt; if the handicap was, say, −7, then the score is only +3. If, say, a score of +15 is needed for conviction, the prosecution still has a long way to go.
(By the way, did you see my reply to your comment about psychological evidence?)
LW ref: Privileging the hypothesis.
You have to explain that Steve was chosen randomly for your example to be right.
People understand counting better than probability. Talk about a group of people and how many of them match. (Starting with smaller numbers might help too.)