I’m pretty confused about how PCR testing can be so bad. Do you have more models/info here you can share?
In particular, I think it might be the case that we’ve done something like overupdate on poorly-done early Chinese PCR. When I looked for data a while back, I only found the early Wuhan stuff, and the company-backed studies claiming 98% or 99% accuracy, neither of which seem trustworthy...
I currently suspect that PCR tests are effective, at least if the patient has grown enough virus to soon be infectious. I’d like to know if this is true. The main beliefs I have here (that may well be false):
The PCR methodology, when done right, should detect the presence of tiny amounts of viral fragments.
The amount of virus needed per unit saliva to infect someone is at least a few orders of magnitude less than the detection threshold for PCR or other amplification techniques.
If my picture is right, I can perhaps still believe in a 50% false negative rate, but I would look to explain that as “you tested them too early in the infection”, and would suspect the false negative rate to be more like 1-5% for a patient that’s shedding enough virus to be infectious.
In short, I don’t really know how it can be as bad as I claim it is. It seems like it should straightforwardly be highly accurate because of your two points: the sensitivity should be at a much lower threshold than the amount needed to infect someone.
Yet, I still believe this. Part of this belief is predicated on the heterogeneous results from studies, which make me think that “default” conditions lead to lots of false negatives and later studies showed much lower false negatives because they adjusted conditions to be more sanitary and less realistic. However, this is just an extrapolation, and I haven’t looked into these studies unfortunately.
The bigger reason for my belief is that I’ve seen several people almost-definitely get COVID and then test negative.
First data point: B was out in public unmasked, got it, then their family got it. 4ish people showed symptoms, one didn’t. 2-3 tested positive for COVID, the others didn’t, including 2 who tested negative 3ish times in a row, using PCR. B was one that tested negative repeatedly. B was notified shortly afterward that the person they were with in public had tested positive for COVID.
Second data point: C went back to university in Sep 2020. Two of their family members visited. Shortly after, C got pretty sick and tested positive for COVID. Then their family members got pretty sick with flu-like symptoms. Both family members went to the doctor after symptoms and tested negative by PCR.
Less-strong data point: D flew on a plane from the Bay late-Feb/early-Mar 2020. They landed, two days later they got sick with cough, maybe more, felt pretty bad, and had an spO2 of 85. They tested negative 2-3 times by PCR.
I heard about these cases because they were fairly close to me. There were maybe 2 other cases as close to me as these, so these represent about half my epistemic exposure to COVID cases on the ground.
I don’t know how to possibly parse the first two cases aside from saying that the PCR tests gave false negatives. You can’t even say “they got the flu”—their family members tested positive for COVID! The best alternative explanations seem truly terrible: there’s a minuscule chance I just got a wildly skewed sample, or I could’ve done a truly abysmal job at noticing some selection effect. So I feel like I basically have to take them at face value. Using data points 1 and 2 only, and adding 2 positives from cases close ot me, the PCR tests gave false negatives roughly 7⁄10 + 2⁄3 + 0/2= 9⁄15 times, ~60% false negative rate, but if you count by person this is only ~40% false negative, which is the better way to look at it to correct for the selection effect of people who test negative getting tested repeatedly. Maybe someone got rapid when they thought they got PCR, and you lower 10% to bring the total to 30% false negative rate by person. But not much more in sample mean.
As I said, I don’t really understand how PCR tests can be this bad. However, it would tie up very neatly if, for example, COVID just didn’t make it to the nose in a lot of patients. Perhaps orders of magnitude more is coughed out of the lungs as an infection vector than is exuded from the nostrils. Or perhaps the efficacy of swabbing varies a ton, or the efficacy of testing—a lot of the negative tests I know about were from Red states, and I can’t help but wonder if the old “getting the results you want to get” effect is striking in another wild circumstance (but of course note the selection bias since I know of more cases in Red states).
And even without knowing how PCR tests can be so bad, I don’t feel like I’m going that much out on a limb when I’m imagining there might be lots of heterogeneity in how they’re done. Even if the best tests are really quite good, if the worse tests have user-error rates of one in five, and these are selected to be the ones more in use where COVID outbreaks are (due to culture or the obvious causality), you could potentially have a lot of people with 20% FNR rates. Also, while I think lots of criticisms that “the lab is different than the real world” are misplaced, COVID tests seem like almost a central case where you’d expect that specific failure mode.
(I probably won’t delve into the papers to try to figure this out, but I would love to hear from anyone else who might have alternative hypotheses about this, or reasons why “COVID doesn’t always go to the nose” shouldn’t be the default hypothesis here.)
Another cool data point! I found a paper from Singapore, Jul 2020, testing tear swabs but incidentally giving a bunch of PCR tests too. I’m much more likely to trust a paper that gives PCR tests incidentally, rather than is directly testing their effectiveness with researcher bias toward better results. By counting up the squares by hand, this paper shows 24⁄108 PCR tests came back negative if I counted correctly: that’s 22% false negative rate (FNR).
Now, for adjustments:
First, these patients were recruited from a hospital. So they obviously have much higher viral load than the average person, so we’d expect higher FNR for the general population. (And we see the expected relationship between viral load and positive results: people with average low Ct values (meaning high viral load) rarely test negative, but those testing negative lots have very high Ct on their positive tests.)
On the other hand, only 2⁄17 patients test negative >50% of the time; a lot of the negatives come near the end of a patient’s sickness or hospital stay. So we don’t see great empirical evidence for the hypothesis that some people are consistent false-negatives. If you take out the negatives-at-the-end effect, there are far fewer false negatives, maybe 5-10%. However, this is basically moot because of the selection effect for the hospitalized as mentioned above. Of course you’ll see hardly any consistent-false-negative-patients in the hospitalized!—the fact you see any macroscopic number of false negatives in the middle of progression is a terrible sign (and, if there were any fully-false-negative patients, we wouldn’t see them anyways! Bad filter).
And we do see the requisite theoretical evidence. Because of the two patients with repeated false negatives and low viral load when positive, we can easily extrapolate that some patients just have slightly lower viral load and test negative consistently.
Overall, there isn’t much easy way to convert this study into “FNR on asymptomatic individuals who get tested”. However, I think if 5-10% of tests on the hospitalized came back negative, that strongly implies more than a 20% FNR on the asymptomatic. I would personally guess that this lends credence toward 10-40% FNR on the symptomatic and 20-80% FNR on asymptomatic. (Lest I double-count evidence, let it be known I’m basing these numbers in part on the above analysis of my personally-known symptomatic individuals with ~40% FNR.)
I believe FNR depends on swabbing (which varies based on equipment and individual doing it), on PCR equipment, and on the patients (e.g. how early are you testing people? age of patients, etc). Then there’s issue of how you get ground-truth which might also contribute to variation in these estimates.
I once looked into the effectiveness of Australia and New Zealand’s quarantine programs to get a sense for this. I think, until recently, basically no infectious cases made it through their 2-week quarantines. Their track records have become more marred since Delta arrived.
For New Zealand, if I recall correctly, basically no community infection clusters were due to quarantine breakthroughs (citation needed!). Of the cases caught with PCR, 80% tested positive on day 3 of quarantine, and the remaining 20% were positive on day 12.
So while some cases might have not been detected, it seems like these didn’t go on to infect others after the 2 weeks. The people getting tested on day 3 could have been infected on their plane flight, or perhaps some days before their flight. I’d guess the median infection was a week old by day 3 of quarantine.
Anyhow, NZ’s numbers seemed to rule out a 50% false-negative rate, because I think their quarantine would have failed if so. They also seem to rule out a 2% false-negative rate, at least for tests done early after infection.
Our current Delta outbreaks are tracable to a limo driver who was not—nor required to be—vaccinated or even masked while transferring travellers from their flight to hotel quarantine.
The main source of our success has been in massively cuts to the number of travellers we allow in, and that has it’s own obvious problems...
In Australia, hotel quarantine has caused one outbreak per 204 infected travellers. Purpose-built facilities are far better, but we only have one (Howard Springs, near Darwin) and the federal government has to date refused to build any more.
But no cases of infections slipping thru the testing, no?
As far as I know none of our leaks have been by releasing an infectious person after a negative test result.
It’s possible for PCR tests to return negative for a very early (low viral load) infection though; that’s why for high-risk travellers we do PCR tests on days −3, 1, 5, 11, and 14 of the quarantine period. For low-risk settings, ie contact tracing, you only need to isolate until you get a negative PCR test result.
I’m pretty confused about how PCR testing can be so bad. Do you have more models/info here you can share?
In particular, I think it might be the case that we’ve done something like overupdate on poorly-done early Chinese PCR. When I looked for data a while back, I only found the early Wuhan stuff, and the company-backed studies claiming 98% or 99% accuracy, neither of which seem trustworthy...
I currently suspect that PCR tests are effective, at least if the patient has grown enough virus to soon be infectious. I’d like to know if this is true. The main beliefs I have here (that may well be false):
The PCR methodology, when done right, should detect the presence of tiny amounts of viral fragments.
The amount of virus needed per unit saliva to infect someone is at least a few orders of magnitude less than the detection threshold for PCR or other amplification techniques.
If my picture is right, I can perhaps still believe in a 50% false negative rate, but I would look to explain that as “you tested them too early in the infection”, and would suspect the false negative rate to be more like 1-5% for a patient that’s shedding enough virus to be infectious.
In short, I don’t really know how it can be as bad as I claim it is. It seems like it should straightforwardly be highly accurate because of your two points: the sensitivity should be at a much lower threshold than the amount needed to infect someone.
Yet, I still believe this. Part of this belief is predicated on the heterogeneous results from studies, which make me think that “default” conditions lead to lots of false negatives and later studies showed much lower false negatives because they adjusted conditions to be more sanitary and less realistic. However, this is just an extrapolation, and I haven’t looked into these studies unfortunately.
The bigger reason for my belief is that I’ve seen several people almost-definitely get COVID and then test negative.
First data point: B was out in public unmasked, got it, then their family got it. 4ish people showed symptoms, one didn’t. 2-3 tested positive for COVID, the others didn’t, including 2 who tested negative 3ish times in a row, using PCR. B was one that tested negative repeatedly. B was notified shortly afterward that the person they were with in public had tested positive for COVID.
Second data point: C went back to university in Sep 2020. Two of their family members visited. Shortly after, C got pretty sick and tested positive for COVID. Then their family members got pretty sick with flu-like symptoms. Both family members went to the doctor after symptoms and tested negative by PCR.
Less-strong data point: D flew on a plane from the Bay late-Feb/early-Mar 2020. They landed, two days later they got sick with cough, maybe more, felt pretty bad, and had an spO2 of 85. They tested negative 2-3 times by PCR.
I heard about these cases because they were fairly close to me. There were maybe 2 other cases as close to me as these, so these represent about half my epistemic exposure to COVID cases on the ground.
I don’t know how to possibly parse the first two cases aside from saying that the PCR tests gave false negatives. You can’t even say “they got the flu”—their family members tested positive for COVID! The best alternative explanations seem truly terrible: there’s a minuscule chance I just got a wildly skewed sample, or I could’ve done a truly abysmal job at noticing some selection effect. So I feel like I basically have to take them at face value. Using data points 1 and 2 only, and adding 2 positives from cases close ot me, the PCR tests gave false negatives roughly 7⁄10 + 2⁄3 + 0/2= 9⁄15 times, ~60% false negative rate, but if you count by person this is only ~40% false negative, which is the better way to look at it to correct for the selection effect of people who test negative getting tested repeatedly. Maybe someone got rapid when they thought they got PCR, and you lower 10% to bring the total to 30% false negative rate by person. But not much more in sample mean.
As I said, I don’t really understand how PCR tests can be this bad. However, it would tie up very neatly if, for example, COVID just didn’t make it to the nose in a lot of patients. Perhaps orders of magnitude more is coughed out of the lungs as an infection vector than is exuded from the nostrils. Or perhaps the efficacy of swabbing varies a ton, or the efficacy of testing—a lot of the negative tests I know about were from Red states, and I can’t help but wonder if the old “getting the results you want to get” effect is striking in another wild circumstance (but of course note the selection bias since I know of more cases in Red states).
And even without knowing how PCR tests can be so bad, I don’t feel like I’m going that much out on a limb when I’m imagining there might be lots of heterogeneity in how they’re done. Even if the best tests are really quite good, if the worse tests have user-error rates of one in five, and these are selected to be the ones more in use where COVID outbreaks are (due to culture or the obvious causality), you could potentially have a lot of people with 20% FNR rates. Also, while I think lots of criticisms that “the lab is different than the real world” are misplaced, COVID tests seem like almost a central case where you’d expect that specific failure mode.
(I probably won’t delve into the papers to try to figure this out, but I would love to hear from anyone else who might have alternative hypotheses about this, or reasons why “COVID doesn’t always go to the nose” shouldn’t be the default hypothesis here.)
Another cool data point! I found a paper from Singapore, Jul 2020, testing tear swabs but incidentally giving a bunch of PCR tests too. I’m much more likely to trust a paper that gives PCR tests incidentally, rather than is directly testing their effectiveness with researcher bias toward better results. By counting up the squares by hand, this paper shows 24⁄108 PCR tests came back negative if I counted correctly: that’s 22% false negative rate (FNR).
Now, for adjustments:
First, these patients were recruited from a hospital. So they obviously have much higher viral load than the average person, so we’d expect higher FNR for the general population. (And we see the expected relationship between viral load and positive results: people with average low Ct values (meaning high viral load) rarely test negative, but those testing negative lots have very high Ct on their positive tests.)
On the other hand, only 2⁄17 patients test negative >50% of the time; a lot of the negatives come near the end of a patient’s sickness or hospital stay. So we don’t see great empirical evidence for the hypothesis that some people are consistent false-negatives. If you take out the negatives-at-the-end effect, there are far fewer false negatives, maybe 5-10%. However, this is basically moot because of the selection effect for the hospitalized as mentioned above. Of course you’ll see hardly any consistent-false-negative-patients in the hospitalized!—the fact you see any macroscopic number of false negatives in the middle of progression is a terrible sign (and, if there were any fully-false-negative patients, we wouldn’t see them anyways! Bad filter).
And we do see the requisite theoretical evidence. Because of the two patients with repeated false negatives and low viral load when positive, we can easily extrapolate that some patients just have slightly lower viral load and test negative consistently.
Overall, there isn’t much easy way to convert this study into “FNR on asymptomatic individuals who get tested”. However, I think if 5-10% of tests on the hospitalized came back negative, that strongly implies more than a 20% FNR on the asymptomatic. I would personally guess that this lends credence toward 10-40% FNR on the symptomatic and 20-80% FNR on asymptomatic. (Lest I double-count evidence, let it be known I’m basing these numbers in part on the above analysis of my personally-known symptomatic individuals with ~40% FNR.)
Also, when I look around, I find charts like these that suggest the claimed false negative rates vary absurdly!
I believe FNR depends on swabbing (which varies based on equipment and individual doing it), on PCR equipment, and on the patients (e.g. how early are you testing people? age of patients, etc). Then there’s issue of how you get ground-truth which might also contribute to variation in these estimates.
Where did this chart come from?
Oops! Source: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0242958
I once looked into the effectiveness of Australia and New Zealand’s quarantine programs to get a sense for this. I think, until recently, basically no infectious cases made it through their 2-week quarantines. Their track records have become more marred since Delta arrived.
For New Zealand, if I recall correctly, basically no community infection clusters were due to quarantine breakthroughs (citation needed!). Of the cases caught with PCR, 80% tested positive on day 3 of quarantine, and the remaining 20% were positive on day 12.
So while some cases might have not been detected, it seems like these didn’t go on to infect others after the 2 weeks. The people getting tested on day 3 could have been infected on their plane flight, or perhaps some days before their flight. I’d guess the median infection was a week old by day 3 of quarantine.
Anyhow, NZ’s numbers seemed to rule out a 50% false-negative rate, because I think their quarantine would have failed if so. They also seem to rule out a 2% false-negative rate, at least for tests done early after infection.
In Australia, hotel quarantine has caused one outbreak per 204 infected travellers. Purpose-built facilities are far better, but we only have one (Howard Springs, near Darwin) and the federal government has to date refused to build any more.
Our current Delta outbreaks are tracable to a limo driver who was not—nor required to be—vaccinated or even masked while transferring travellers from their flight to hotel quarantine.
The main source of our success has been in massively cuts to the number of travellers we allow in, and that has it’s own obvious problems...
But no cases of infections slipping thru the testing, no?
As far as I know none of our leaks have been by releasing an infectious person after a negative test result.
It’s possible for PCR tests to return negative for a very early (low viral load) infection though; that’s why for high-risk travellers we do PCR tests on days −3, 1, 5, 11, and 14 of the quarantine period. For low-risk settings, ie contact tracing, you only need to isolate until you get a negative PCR test result.