A breakdown of priors and posteriors - an example from medicine

Link post

This post is lifted (with edits) from my comment on a thread on Reddit that talked about a blood test that detected amyloid plaque, a potential precursor to Alzheimers. A top-level commenter felt that the fact that the test couldn’t tell for certain whether someone had Alzheimers meant that it was useless.

“Not everyone with amyloid in their brains will turn out to have dementia, and not everyone who has dementia will be found to have amyloid in their brains.”

So what is the point of this article again? To get a load of clicks I assume, cos there’s no real content here.

I attempted to correct that misconception.

(If anyone knows how to do the whole thing with odds ratios rather than probabilities, please let me know! We can’t update as normal here.)

tl;dr: No test is ever 100% certain, but it can still constrain your probabilities, making you more certain than you were before. Not everyone who is outside during a thunderstorm gets hit by lightning, and not everyone who gets hit by lightning was outside during a thunderstorm. Going outside in a thunderstorm is still much more dangerous than staying inside.

Notation: $P (d e m e n t i a | a m y l o i d)$ is read as the probability of having dementia given that you have amyloid. To make things fit in a LessWrong post, I’ll abbreviate having dementia to $D$ , and having amyloid to $A$ . A positive test result is $+$ , and a negative test result is $-$ . Finally, a negation is $\neg$ , so $\neg A$ would be no amyloid.

$O (D : \neg D)$ , read as the odds of dementia to no dementia, is the odds ratio that $D$ is true compared to the odds ratio that $D$ is false. $O (D : \neg D) = 3 : 1$ means that it’s 3 times as likely that somebody has dementia than that they don’t. In general, odds doesn’t say anything about the magnitude of the probabilities, so they could be small, like 3% and 1%, or big, like 60% and 20%. (Here, of course, $P (D) + P (\neg D) = 1$ because the two choices represent all possibilities, so the probabilities have to be 75% and 25% respectively.)

Imagine you have ten thousand people in the right demographics. How about these (the numbers are made up):

Dementia status	Has amyloid	Does not have amyloid	Total
Has dementia	1000	300	1300
Does not have dementia	100	8600	8700
Total	1100	8900	10000

So, a person chosen at random from that population has a $P (D) = 13 %$ chance of having dementia, and a $P (\neg D) = 87 %$ chance of not having dementia. Their odds ratio for dementia is $O (D : \neg D) = 13 : 87 \approx 1 : 6.7$ . They’re almost 7 times as likely to have no dementia than they are to have it.

Now, let’s say that if you have amyloid, the test will say that you have amyloid 95% of the time, while if you do not have amyloid, the test will say that you have amyloid 10% of the time. In other words, $P (+ | A) = 95 %$ , and $P (+ | \neg A) = 10 %$ . The probability of negative results is just 100% minus the probability of the corresponding positive result. (The actual status is on the top row, and the test response is on the left.)

Test result	Has amyloid	Does not have amyloid
Positive test	95%	10%
Negative test	5%	90%

Now, let’s say that you take the test and it says that you do have amyloid. What can we say about your probability of dementia? Can we do better than that 13%?

If the test is positive, one of four things might be true:

The test is correct, and you do have amyloid, $P (A | +)$ . You’re one of those people who has dementia and amyloid, $(D, A)$ .
The test is correct, and you do have amyloid, $P (A | +)$ . However, you’re one of those people who has amyloid despite not having dementia, $(\neg D, A)$ .
The test is incorrect, and you do not have amyloid, $P (\neg A | +)$ . However, you’re one of those people who has dementia despite not having amyloid, $(D, \neg A)$ .
The test is incorrect, and you do not have amyloid, $P (\neg A | +)$ . You’re one of the majority who have neither dementia nor amyloid, $(\neg D, \neg A)$ .

Now, we can find the probability that you do have dementia if you get a positive test result:

$P (D | +) = P (D | A) \times P (A | +) + P (D | \neg A) \times P (\neg A | +)$

$P (D | A)$ here is simply the 1000 people with dementia out of the 1100 with amyloid, or $10 / 11$ . Similarly, $P (D | \neg A)$ is 300 people with dementia out of the 8900 people without amyloid, or $3 / 89$ .

What about the $P (A | +)$ and $P (\neg A | +)$ ? Let’s calculate the odds ratio for that first.

The prior odds ratio, from the distribution in the population, is $11 : 89$ . The Bayes Factor for a positive test is 95% for people with amyloid vs 10% for people without, or $95 : 10 = 9.5$ . So, $O (A : \neg A | +) = 11 : 89 \times 9.5 = 104.5 : 89 \approx 54 : 46$ . Changing that into probabilities gives us $P (A | +) = 54 %$ and $P (\neg A | +) = 46 %$ .

Bringing it all together, we get:

$P (D | +) = 10 / 11 \times 0.54 + 3 / 89 \times 0.46 = 50.6 %$

We can also find the probability that you have dementia if you get a negative test result. The Bayes Factor here is $5 : 90 = 1 : 18$ , and $O (A : \neg A | -) = 11 : 89 \times 1 : 18 = 11 : 1602$ . Therefore, $P (A | -) = 0.68 %$ and $P (\neg A | -) = 99.32 %$ .

$P (D | -) = P (D | A) \times P (A | -) + P (D | \neg A) \times P (\neg A | -)$

Substitute the numbers in, and we get:

$P (D | -) = 10 / 11 \times 0.0068 + 3 / 89 \times 0.9932 = 3.97 %$

So, we end up with one final table. The probability that someone doesn’t have dementia given a test result is just 100% minus the probability of having dementia. (What the test says is on top, and whether you actually have dementia is on the left.)

Dementia status	Positive test	Negative test
Has dementia	50.6%	4.0%
Does not have dementia	49.4%	96.0%

In other words, a random person would have a 13% chance of having dementia, $P (D)$ . However, if your test is positive for amyloid, we adjust to a 50.6% chance that you have dementia, $P (D | +)$ , and a 49.4% chance that you don’t, $P (\neg D | +)$ . This is considering the fact that some people have amyloid but no dementia, the fact that some people have no amyloid and dementia, and the fact that the test can just be wrong.

On the other hand, if the test is negative, it would go down from 13% to 4%, but that still isn’t zero. It could be that you were just unlucky and the test didn’t register your amyloid.

If the test is positive, you’re about as certain that you have dementia than that you don’t have dementia, since $O (D : \neg D) = 50.7 : 49.3 \approx 1.01 : 1$ . You are now almost 7 times more sure that you have dementia than you were before. ( $\frac{50.7}{49.3} \times \frac{87}{13} \approx 6.9$ )

If the test is negative, you’re 24 times more certain that you don’t have dementia than that you do, since $O (D : \neg D) = 4 : 96 = 1 : 24$ . Compared to your prior belief, you’re more than 3.5 times as sure that you don’t have dementia than you were before. ( $\frac{96}{4} \times \frac{13}{87} \approx 3.6$ ) (Notice how we flipped the fractions since we’re testing the negative case.)

I wouldn’t consider a negative case that big of a deal, but the positive one certainly warrants further testing. If you use another test, which checks for things other than amyloid, or for amyloid using a different mechanism, you’ll be much more sure of your result because your starting probability would now be the 50.6%, not the original 13%. (The order of the tests doesn’t matter, but a first screening would probably be with a test that is cheap and/or quick.)

edit 1: Fixed the explanation of odds ratios. Thanks to Joachim Bartosik for pointing it out.

edit 2: Fixed the calculation of the posterior, and added step-by-step calculations.

A breakdown of priors and posteriors—an example from medicine