The Case for Extreme Vaccine Effectiveness
I owe tremendous acknowledgments to Kelsey Piper, Oliver Habryka, Greg Lewis, and Ben Shaya. This post is built on their arguments and feedback (though I may have misunderstood them).
Update, May 13
I first wrote this post before investigating the impact of covid variants on vaccine effectiveness, listing the topic as a major caveat to my conclusions. I have now spent enough time (not that much, honestly) looking into variants that I have a tentative position I’m acting on for now.
My moderately confident conclusion is that the current spread of variants in the US barely impacts vaccine effectiveness. The Pfizer vaccine is reported to be 85% as effective against the feared B.1.351 variant (South African) as it is against B.1.1.7 (UK). Assuming that other variants are no more resistant than B.1.351 on average (a reasonable assumption) and that presently variants are no more than 25% of Covid cases (in Alameda and San Francisco). The net effect is 0.25*0.85 + 0.75*1.0 = 0.9625. In other words, vaccines still have 96% of the effect they would if B.1.1.7 were the only variant.
Plus, that tiny reduction of vaccine effectiveness is dwarfed by the falling background prevalence of Covid. When I first wrote this post, Alameda and San Francisco were at 0.1-0.15%; now they’re at ~0.05%. The same for New York and the United Kingdom.
Although relaxing of restrictions might reverse this, right now, Covid-risk is very, very low in the Bay Area and many parts of the US.
All updates/changelog can be viewed here.
I plead before the Master of Cost-Benefit Ratios. “All year and longer I have followed your dictates. Please, Master, can I burn my microCovid spreadsheets? Can I bury my masks? Pour out my hand sanitizer as a libation to you? Please, I beseech thee.”
“Well, how good is your vaccine?” responds the Master.
“Quite good!” I beg. “We’ve all heard the numbers, 90-95%. Even MicroCOVID.org has made it official: a 10x reduction for Pfizer and Moderna!”
The Master of Cost-Benefit Ratio shakes his head. “It helps, it definitely helps, but don’t throw out that spreadsheet just yet. One meal at a crowded restaurant is enough to give even a vaccinated person hundreds of microCovids. Not to mention that your local prevalence could change by a factor of 5 in the next month or two, and that’d be half the gains from this vaccine of yours!”
I whimper. “But what if . . . what if vaccines were way better than 10x? What about a 100x reduction in the risks from COVID-19?”
He smiles. “Then we could go back to talking about how fast you like to drive.”
In its most extreme form, I have heard it claimed that the vaccines provide 10x reduction against regular Covid, 100x against severe Covid, and 1000x against death. That is, for each rough increase in severity, you get 10x more protection.
This makes sense if we think of Covid as some kind of “state transition” model where there’s a certain chance of moving from lesser to more severe states, and vaccines reduce the likelihood at each stage.
I think 10x at multiple stages is too much. By the time you’re at 1000x reduction, model uncertainty is probably dominating. I feel more comfortable positing up to 100x, maybe 500x reduction. I dunno.
There is a more limited claim of extreme vaccine effectiveness that I will defend today:
In the case of the Pfizer vaccine (and likely Moderna too), the effectiveness in young healthy people is 99% against baseline symptomatic infection, or close to it.
We can reasonably expect the effectiveness of the vaccine against more severe cases of Covid to be greater than effectiveness against milder cases of Covid.
(Maybe it’s 2x more effective against severe-Covid and 3x more effective against death compared to just getting it at all. Something like that, it doesn’t have to be 10x– it’d still be a big deal because more severe outcomes are where most of the disutility lies.)
It’s a very simple argument, really. First, the data very clearly suggests effectiveness of ~99% for young people, with nice tight confidence intervals. Second, across all the data we see trends of increasing effectiveness against increasing severity, granted that the confidence intervals are wide in some cases. Third, a very reasonable (imo) mechanistic model supports this interpretation of the data.
The 1.2 Million-Person Pfizer Israeli Observational Study
This observational study matched ~600k vaccinated people 1:1 with ~600k demographically similar controls. It covered the period December 20 to February 1. As far as I know, it is by far the largest Covid-19 vaccine study published to date. The other studies are clinical trials with sample sizes on the order of 20k-40k, and some other observational studies, typically with healthcare workers, in the single-digit thousands.
Why it’s not as big as it sounds
Before we get to looking at data, I think it’s important to note why uncertainty remains despite the huge N. To start with, the outcomes are all quite rare. Eyeballing it, Israel had a Covid-19 prevalence of ~0.5% during the study period. Out of a million people, a few thousand might be expected to actually catch Covid. Of that few thousand, only dozens or hundreds will progress to more severe forms of Covid. When sample sizes are in the dozens, confidence intervals are wide.
The authors report:
During a mean follow-up of 15 days (interquartile range, 5 to 25), 10,561 infections [6,101 control vs 4,460 vaccinated] were documented . . . of which 5996 [2494 vs 2,071](57%) were symptomatic Covid-19 illness, 369 required hospitalization [259 vs 110], 229 were severe cases of Covid-19 [174 vs 55], and 41 resulted in death [32 vs 9].
See Appendix B for complete breakdown of outcomes by period and vaccination status.
What’s more, those numbers are for the entire study period (44 days). Only in a subset of days had participants been vaccinated long enough for it to be a real test. Nominally it is a 1.2 million person study, but practically, when you’re looking at results 2 weeks after the first dose (~day 14) or one week after the second dose (~day 28), the numbers are much lower. ~80% lower.
All that to say, sample sizes aren’t as big as they sound. Well, let’s look at the results. This is the main outcome table. Definitions in Appendix A.
It’s a bit hard to track trends formatted like that, so here’s an equivalent graph:
Middle: 21-27 days after 1st dose;
Right: 7 days after 2nd dose until end of follow-up
To me, the headline result is that 2nd-dose + 7-days, vaccine effectiveness against Symptomatic Illness is 94% (87-98). Pretty good! Also, efficacy clearly rises from the earlier to later periods after vaccine administration.
Unfortunately, we don’t see efficacy improvements moving to the right on the rightmost graph (more severe outcomes), counter to claims of extreme vaccine effectiveness. We do see upwards-right trends in the earlier periods (left and middle graphs).
Well, 2 out of 3 ain’t bad! Too bad the last one is the one we care about most.
It’s okay. I’ve got more. The astute reader will have noted that the above graphs have “subgroup = Full” in their title. The study made available endpoints (outcomes) for multiple subgroups. Below are some of them. (Here are ALL OF THEM.)
Since there was data for it, I also added in the “2nd dose and 6 days after” period.
Many of the points are missing. The authors did not compute vaccine effectiveness if the control and vaccine group combined did not have 10 or more instances of an outcome. For example, the Age 16-39 subgroup did not have 10 instances of hospitalization, severe-Covid, or death for almost all of the time periods.
In cases where there are 10 instances of an outcome in the control group but 0 in the vaccine group, a value is reported without confidence intervals, e.g. the dots in the Females Subgroup, 2nd-2nd+6 period.
The up-and-to-right shape of the graphs persists across subgroups, except for Males and when the values are already maxing out near 100% (the right-most graphs). Overall, I think this is suggestive of the general trend that vaccines are more effective against progressively more severe outcomes. I also suspect that an uncertainty modeling that took account of the not uncorrelated neighboring values would shrink the error bars beyond the naive bootstrap method.
I’ll comment more on why the flat/missing trend in the rightmost graph doesn’t bother me much, beyond the fact that sample size means it’s hard for it to show much at all.
Also! If you prefer Tables, here’s the top half from the paper itself corresponding to the above graphs:
As someone falling in the Age 16-39 subpopulation, I’m quite pleased to see 99% effectiveness against Symptomatic Infection, with a nice tight 96-100 confidence interval. This is higher than I’d had anyone cite to me, and is approximately a 5x increase in how effective I believe my Pfizer vaccine to be. That’s even before we get to more severe outcomes.
So why is the absence of data showing increasing efficacy not evidence of absence?
Because we expect to see data that looks like this even in worlds where the vaccine is 100% effective (at least for all vaguely healthy people). To be evidence against something, it has to be less common in worlds where that thing is true, and that’s not the case here.
Why would we see these numbers with a 100% effective vaccine?
“Saturation” and “noise”
When you have 100 true positives and 3 false positives, the false positives aren’t such a big deal. When you have 0 true positives and 3 false positives, the false positives can change the entire picture.
I argue this is very likely what is going on with Covid-vaccine effectiveness, above and beyond sample sizes.
Consider that PCR Covid-19 tests have both a false negative and false positive rate (FPR). According to this random site I found by Googling that looks legit enough, the FPR for Covid-19 tests is between 0.2% and 0.9%. Let’s choose a point estimate of 1% for the FPR to be safe, but run it twice for every case to compensate. So now our FPR is 0.01%
Let’s now imagine using this test on a 99% effective vaccine (the same argument holds for 99.9% and 99.99% even more so). We run an RCT with 100,000 people receiving the vaccine and 100,000 receiving placebo. Covid-19 prevalence is 0.1% in the region our hypothetical test is running.
99 people from the control group catch actual Covid and receive positive test results (we lose one to a realistic false-negative rate of 10%, run twice to become 1%) plus 0.01% false positives for a total of 109. From the treatment group with 99% effective vaccine, we get 1 true-positive and 10 false-positive test results. Our final effectiveness estimate is 1 − 11⁄109 = 90%
90%! And that’s from what is in truth a 99% effective vaccine. The control is mostly unaffected by the noise (109 vs actual 100) but the treatment is enormously changed. Instead of 1, it’s 11.
The greatest noise of all is selection effects
Never mind fluke false positive tests, on priors we have reason to suspect that the people who are still getting severe Covid despite being vaccinated or even hospitalized are very likely not like you. Why isn’t the number of vaccinated people who ended up in critical condition zero?
Because within the Israel Observation study, the vaccinated group contains some very sick people. (In a half-million person group without exclusion criteria against it there simply will be, and that’s before the fact that we know many people are elderly and many explicitly meet risk criteria such as cancer, obesity, pulmonary disease, Type 2 diabetes, etc. See Table 1. Demographic and Clinical Characteristics of Vaccinated Persons and Unvaccinated Controls at Baseline.)
Vaccines work by stimulating an immune response. If your immune system is in tatters, unable to manufacture healthy antibodies or something, your vaccine might not do much. You might fare little better than the unvaccinated.
There’s actually a term for immune system failure in the elderly: immunosenescence. A couple of papers on that topic: Immunosenescence and vaccine failure in the elderly (2009) and Immunosenescence: A systems-level overview of immune cell biology and strategies for improving vaccine responses (2019)
Quoting the abstract from the first one:
An age-related decline in immune responses in the elderly results in greater susceptibility to infection and reduced responses to vaccination. This decline in immune function affects both innate and adaptive immune systems . . . Essential features of immunosenescence include: reduced natural killer cell cytotoxicity on a per cell basis; reduced number and function of dendritic cells in blood; decreased pools of naive T and B cells; and increases in the number of memory and effector T and B cells . . . Consequently, vaccine responsiveness is compromised in the elderly, especially frail patients . . . In the future, the development and use of markers of immunosenescence to identify patients who may have impaired responses to vaccination, as well as the use of end-points other than antibody titers to assess vaccine efficacy, may help to reduce morbidity and mortality due to infections in the elderly.
The last line there suggested antibodies can be present even when a vaccine overall is not protective against disease.
Don’t get old kids, it’s bad for you.
When I see that the treatment group has lower but not zero severe cases than the control group, I assume it’s coming from the very ill, the immunocompromised and immunosenescent.
I am pretty confident that I am not in those groups. I’m pretty sure my vaccine elicited some real response (I had some side effects, not terrible, but some). When I see the effectiveness numbers showing globally that there’s still some chance of really bad outcomes, I adjust them downwards because they were very likely not happening to people with remotely my level of health.
To invoke the math of the previous section regarding false positives, the immunocompromised might be only 10% to the number of severe Covid patients in the control group (because their prevalence is low), but after the filter/selection effect of vaccination, they make up 95% of severe cases in treatment group.
Here’s a diagram for good measure:
This is a causal diagram. Arrows indicate the direction of causality but not the specific causal relationship. (Having a poor immune system, e.g. because you’re older, might make you more likely to be vaccinated, but of course, in our study we’re conditioning on vaccination status, so it doesn’t matter.)
Since we didn’t condition on immune system health, we can’t expect that a naive interpretation of the table of efficacies (above) tells us the true relationship between vaccines and outcomes.
Well, we approximately can condition. The vaccine efficacy is higher for the Age 16-39yr subgroup (99%, 96-100) vs the entire population (94%, 87-98). It’s not just that younger people get less Covid, but that the vaccine worked better on them.
The Age 16-39yr subgroup didn’t have zero cases of symptomatic disease (it might have zero of hospitalization, etc), but as above, I’m guessing those were almost all people who knowable had weak immune systems within the overall healthier group too.
Priors & Trends
In the above section I argued two things:
that we see something of a trend towards increasing effectiveness with increasing severity in the Pfizer Israeli mass study
Reasons why even if the vaccine was 99.9% effective, we would expect to observe effectiveness data that is lower than this.
In this section, I want to offer 1) a plausible mechanistic model for why this should be true, 2) further indications of increasing effectiveness from other trials and studies.
Shifting the Distribution of “Infection”/Viral Load
Epistemic status: I’m not a biology/virology/immunology person and this feels kinda hand-wavy to me.
This post started with a table that lists a progression of discrete outcomes: Documented Infection, Symptomatic, Hospitalization, Severe Disease, Death . . . and I’ve been referring to them since. Obviously, the underlying biological reality isn’t quite as discrete as that.
It’s probably more something like there’s a continuous value of how infected you are, and the higher that value gets, the worse your condition will be.
How infected you are is probably a fuzzy thing in reality, but viral load might be an adequate proxy. It’s been documented that viral load varies together with Covid-19 severity. See SARS-CoV-2 viral load is associated with increased disease severity and mortality and Saliva viral load is a dynamic unifying correlate of COVID-19 severity and mortality
The graphs in the second paper particularly make this point.
Presumably, your viral load is the result of competition between the virus replicating and your immune system fighting it. Vaccination gives a significant boost to your immune system. (Cf the oft-cited claim about “4x lower” viral load in vaccinated people)
!! To engage in some inexpert armchair speculation, I’d guess that in the virus-immune system race, about x% of the time the virus gets the upper hand with an infected person and makes them symptomatic (~Level 1), and then in x% of cases where it got to Level 1, the virus wins out again to progress to Level 2 before the immune system can stop it. The virus progresses two levels x^2% of the time. If x is 20%, then overall 4% to get to two levels worse.
At each level, the virus only has an x% of winning out and progressing to the next level. Alternatively, in each time period, there’s some chance, y%, that the immune system will catch up and win.
In cases where someone has a weak immune system (high x%, small y%), increasing levels of case severity aren’t much less likely than earlier ones. You might get an approx flat effectiveness curve.
But suppose that someone is vaccinated and has a real boost to their immune system. Intuitively, I’d reckon they’re now at x/5% for each stage. For the virus to progress two levels, it’s (x/5)^2, or 0.16% when x=20%.
Maybe it’s a factor 2 or 3 instead of 5, but either way, it’d be a compounding effect. More severity means the virus has to replicate more times, which is more time for the immune system to catch up and beat it, ergo less chance for it to get that bad.
It’s tough being a dude
Incidentally, Silva et al (2021) who provided the graphs of viral load immediately above, also had this to say re male vs female:
The difference in viral loads lines up with the Male subgroup having worse vaccine efficacy than the Female subgroup: 88% (71-98) vs 96% (90-100) against symptomatic infections, 2nd dose+7 . It is also the case that across the world,women live longer and die less often from cardiovascular disease, cancer, diabetes, and chronic respiratory disease (Our World in Data).
Johnson & Johnson & Friends
Although not an mRNA vaccine like today’s star, Pfizer’s BNT162b2, the J&J clinical trial tracked multiple endpoints that show our hoped-for trend.
~40,000 participants (treatment and control)
65% of cohort is 18-59yr
The design/reporting of the J&J clinical trial differs from others, particularly the large Pfizer observational study. The time periods and outcomes are defined differently.
To avoid copying multiple tables, I’ve extracted the numbers as I’ve understood them.
Onset at Least 14 Days | Onset at Least 28 Days | |||||
|---|---|---|---|---|---|---|
| Treatment | Placebo | VE% (95% CI) | Treatment | Placebo | VE% (95% CI) | |
| Moderate to Severe, 18-59 yrs | 95 | 260 | 63.7% (53.9-71.6) | 66 | 193 | 66.1% (53.3--75.8) |
| Moderate to Severe, >=60 years | 21 | 88 | 76.3% (61.6-86.0) | 14 | 41 | 66.2% (36.7-83.0) |
| Severe /Critical, 18-59 yrs | 12 | 52 | 76.9% (56.2-88.8) | 5 | 33 | 85% (61.2-95.4) |
| Severe /Critical, >= 60 years | 7 | 28 | 75.1% (41.7-90.8) | 3 | 15 | 80.2% (30-96.3) |
| Requiring Medical Intervention | 2 | 14 | 85.7% (37.8-98.4) | 0 | 7 | 100% (31.1-100) |
Moderate Covid-19: Positive test, any 1 really bad symptom or 2 of regular symptoms like fever, sore throat, cough
Severe/Critical: 3+ regular symptoms or things like respiratory rate ≥30 breaths/minute, heart rate ≥125 beats/minute, oxygen saturation (SpO2) ≤93%, shock, admission to ICU, death
Requiring Medical Intervention: hospitalization, ICU admission, mechanical ventilation, and/or ECMO.
There were deaths in the J&J study, all within the placebo group:
Using a somewhat different formula, the authors also report on interim asymptomatic results. They present four different operationalizations of which I choose two, the ones with the highest and lowest efficacy after 29 days. See Table 20 for further detail.
Day 1-Day 29 | After Day 29 | |||||
|---|---|---|---|---|---|---|
| Treatment | Placebo | VE% (95% CI) | Treatment | Placebo | VE% (95% CI) | |
| FAS seronegative at baseline, +PCR and/or serology and did not show signs and symptoms | 159 | 182 | 12.5% (-8.9-29.7) | 22 | 54 | 59.7% (32.8-76.6) |
| Seroconverted without previous symptoms | 84 | 180 | 22.6% (-3.9-42.5) | 10 | 37 | 74.2% (47.88-88.6) |
Despite the J&J trial using overlapping criteria, both explicitly lumping things and in their criteria, we see the progression we’d expect to see if vaccines work better to prevent worse outcomes than they do milder ones. Modulo confidence intervals, that is.
While efficacy against moderate to severe Covid-19 is 65% (for 18-59 years old), it jumps to 85% for severe alone, granted the overlap in confidence intervals. 0 cases in the vaccine group fell into the Requiring Medical Intervention endpoint, compared to 7 with placebo. It’s as good as we could hope to see with this data.
However, there isn’t a clear jump between “asymptomatic” and “moderate to severe”. Partly because the operationalization isn’t clear (59.7% to 66% is a jump) but there are still wide error bars. The After Day 29 antibody test was conducted at Day 71 and had only been completed for ~30% of participants at the time of publication.
On net, I think the overall endpoint trend lines up with increasing vaccine efficacy against more severe outcomes, but the asymptomatic vs symptomatic doesn’t clearly show it, but part of that is I don’t really understand the groups or how to interpret them.
Let’s get clinical, clinical
Moderna Clinical Trial
The Moderna Phase III Trial tracked Covid and severe Covid but no other end points. There were no severe Covid or deaths in the vaccine group, but 30 severe Covid cases in control and one death. Doesn’t let you compute a precise value, but is consistent with the vaccines being very good.
59% of cohort is aged 18-64 yr and not in any risk categories
Further breakdown of participant characteristics in Table 1
Asymptomatic Infections and Hospitalizations were not tracked.
Table S13 in the appendix provides a very detailed breakdown of symptoms. No severe symptoms at all occured to participants in the treatment group, granted that very few occurred in the control group either.
Pfizer Clinical Trial
58% of cohort is aged 16-55yr, median 52yr, range 16-89
35% have BMI >= 30
After the second dose, there 5 cases of severe-Covid in the control group and 1 in treatment, no reported deaths.
It’s nice (for me) to note that the vaccine efficacy, and particularly the confidence intervals, are higher for the 16-55yr group. 95.6% with CI 89.4%-98.6%. Doesn’t prove the main point, but is in line with the immunosenescence model.
No Covid deaths are reported in either placebo or vaccine groups. Table S5 from the appendix least severe-Covid outcomes. A total of 9 after the first dose in control, 1 in the vaccine group for a 89% reduction with confidence interval between 20.1%-99.7%. If we break it down to different time periods (before/after 1st/2nd dose), we end up with confidence intervals (-3800% to 100%). Yes, maybe taking the vaccine will increase your chance of severe-Covid by 39x!
As expected, the data doesn’t show that the reduction in severe-Covid is greater than lesser-Covid, but it also they doesn’t show the opposite either.
As described above, we have to note the greatly reduced sample size in the later periods.
Evidence of Increased Asymptomatic/Symptomatic Ratios
This was already shown in the mass Pfizer study, but several other sources indicate the ratio of asymptomatic-to-symptomatic cases is increased for vaccinated people. In other words, vaccination works better against symptomatic Covid (more severe) than asymptomatic Covid (less severe). This is at earlier side of “severity”, compared to hospitalization, severe-Covid, and death, but it suggests the same trend–plus there’s more data than when looking at more severe outcomes.
I’ve copied most of the numbers here form MicroCOVID.org, see their analysis for calculation detail.
Unvaccinated | Vaccinated | |||
|---|---|---|---|---|
Symptomatic | % Asymptomatic | Symptomatic | % Asymptomatic | |
| Mass Pfizer Study, Day 28+ | 210 vs 191 | 48% | 31 vs 59 | 65% |
185 vs 37 | 17% | 11 vs 11 | 50% | |
| J&J | 351 vs 182 | 34% | 117 vs 159 | 57% |
| AstraZeneca | 248 vs 73 | 23% | 84 vs 57 | 40% |
Pfizer Asymptomatic percentage goes from 48%->65%, Moderna from 17%->50%, J&J from 34%-57%, AstraZeneca 23%->40%
I expect the very different absolute numbers to come from the widely varying study methodologies as much as differences between the vaccines. (AstraZeneca uses home tests, for example.)
I didn’t exhaustively look through all possible sources of asymptomatic vs symptomatic efficacy. I would be very interested if someone had a credible source not showing this trend.
I also didn’t scrutinize these calculations much, so I wouldn’t be that surprised if it turned out there were deep flaws that undermine the trend seen here.
It gets better
If we go back to the big Israeli Pfizer observational study, we see increasing vaccine effectiveness as more time passes since first/second dose. Unfortunately, the study didn’t have enough time/data to show us things two weeks after the 2nd dose.
Fortunately, there was a follow-up. On March 11, Pfizer/Israel Ministry of Health made the following press release:
Findings from the analysis were derived from de-identified aggregate Israel MoH surveillance data collected between January 17 and March 6, 2021, when the Pfizer-BioNTech COVID-19 Vaccine was the only vaccine available in the country and when the more transmissible B.1.1.7 variant of SARS-CoV-2 (formerly referred to as the U.K. variant) was the dominant strain. Vaccine effectiveness was at least 97% against symptomatic COVID-19 cases, hospitalizations, severe and critical hospitalizations, and deaths. Furthermore, the analysis found a vaccine effectiveness of 94% against asymptomatic SARS-CoV-2 infections. For all outcomes, vaccine effectiveness was measured from two weeks after the second dose.
The lack of actual paper makes this a little harder to interpret, but I don’t find it surprising given that (1) at this later date in their roll-out, an even greater proportion of people will be young and healthy, (2) this data is only counting two weeks after the second dose, whereas the previous large observational study only had a “7 days after 2nd dose until end of follow-up” (maximum of day 28 to 44).
And, of no small significance, the Pfizer vaccine appears fully effective against the UK variant. (Yay!!)
If the vaccine is showing 97+% amongst everyone, I would expect that’s at least as true when you filter for younger/healthier people and filter out those with comorbidities.
What I believe
I believe that what I wrote above supports my initial assertions:
In the case of the Pfizer vaccine (and likely Moderna too), the effectiveness in young healthy people is 99% or close to it.
That we can reasonably expect the effectiveness of the vaccine against more severe Covid to be greater than effectiveness against milder cases of Covid.
The initial Pfizer mass study has 99% (96-100) for the age 16-39yr group, and the subsequent follow-up gives 97% for everyone. At baseline for symptomatic cases, we’re talking 30-100x reductions, which is hella good.
Further, across multiple studies, vaccines, and outcomes we see trends of increasing effectiveness against progressively severe outcomes. In some cases, we don’t definitively see it, but that’s easily attributable to lack of sample size and inherent limitations in methodology due to noise and selection effects.
If we’re talking 99% against symptomatic cases (100x reduction), then I think it’s reasonable to expect at least that for hospitalization, 99.5% (200x). Hence the title, extreme vaccine effectiveness.
What about J&J (and AstraZeneca)? Granted, The effectiveness numbers for J&J look lower than for Pfizer and Moderna, but I think they’re higher than MicroCOVID.org’s numbers imply. First, we get 85% (61.2-95.4) effectiveness against Severe/Critical in the 18-59yr subgroup. That number matters more. Second, that is a very wide age range. I would bet that restricting it to 18-39 would show an improvement relevant to most of those reading this. Lastly, I suspect all the factors mentioned above (selection effects, noise/saturation) to affect it and make the result lower than it would be otherwise.
On net, J&J might not be quite as extremely effective as the mRNA vaccines, but it’s no pushover either.
AstraZeneca isn’t on offer in the Bay, and was recently abandoned in my home country of Australia too, so I apologize for not examining it.
Tell me where I’m wrong
I want the case I’ve made today to be true, but even more than that I want to believe true things (and I certainly don’t want people to believe false things because of me). If you think any of this is wrong. PLEASE SAY SO.
Many thanks!
Objections
Maybe the vaccine and control groups are different?
The Mass Israeli observation study is not an RCT. Instead, it’s looking at their database of vaccinated people and then trying to match each of them to someone who’s mostly the same except not vaccinated.
This might be hard to do. The kinds of people who seek out vaccines (“health-seeking” behavior) are probably a bit different from those who haven’t gotten around to it. Maybe they wear better masks (or masks at all) on the hand, and maybe they risk-compensate on the other.
The authors attempt to control for this by 1:1 matching vaccinated people and controls on multiple characteristics. They cite as evidence this worked the fact the Covid cases among vaccinated and non-vaccinated groups are the same before Day 12 (before the vaccine really starts working), and explain the initial gap in the first couple of days as a selection effect, because vaccinated people only choose to get vaccinated when they’re feeling well.
I didn’t look at the data especially long, but I’m not sure what to make of it. If vaccinated people are more cautious, even after being vaccinated, this means vaccines are less effective than we’d otherwise think. If they risk-compensate, that means vaccines are more effective than what we observe.
I’m not sure how this should net out, but I do think it should widen confidence intervals somewhat.
I also expect the effect to get stronger over time in Israel, as the people who remain unvaccinated and serve as controls (in observational studies) who least cared about Covid and getting vaccinated. So...maybe that explains the later 97% effectiveness result.
Ben Shaya’s Thoughts
Ben Shaya, one of the people responsible for MicroCOVID.org’s models, has been kindly taking time to discuss the topic of vaccine effectiveness with me.
Here wrote a document arguing Why I think vaccines don’t bring the chance of severe COVID to 0. While that’s a stronger claim than I would make, his arguments and models are still valuable when thinking about the topic generally.
Read them all in his doc, but I’ll highlight one that I found very relevant to mechanistic models of Covid severity:
There’s is a bit of mechanistic nuance—COVID tends to start with an upper respiratory tract infection—that’s when nasal swabs work, and then moves to the lower respiratory tract (lungs) - which is where is screws over your blood oxygen. There’s research suggesting that the immune response of the lining of your windpipe is what determines whether the virus reaches your lungs: https://www.biorxiv.org/content/10.1101/2021.02.20.431155v1.abstract?%3Fcollection=
That is to say, there isn’t a single “immune response” that determines how bad your infection is; immune response in your upper respiratory tract determines if you get COVID while immune response in your lungs and windpipe determine whether you get a severe reaction. Immune response in your blood correlates with these, but it’s not the same as either.
(this comes from Riley Drake, who is a virologists and one of the authors on the paper, who strongly cautioned against reading too deeply into antibody concentrations as a proxy for immunity)
When we treat these three immune systems as separate, we see there are at least 3 hidden models—upper respiratory response, lower respiratory response, and blood response. Of the 3, the side effects to the vaccine only reflect the blood response. The overall efficacy of the vaccine only reflects the upper response.
Further, we know that humans have significant heterogeneity in all thee of these responses, since only some people get severe covid, and some people get lung damage from mild covid, and some people get exposed to COVID and don’t contract the virus at all.
Note further that, assuming a strong immune response protects you is also not necessarily true; people with very strong response can cause cytokine storms (which we now know how to handle, but will land you in a hospital).
This gets at more gears in immune response and is the kind of thing that can expose where simple state transition models and immune response as a single thing don’t hold up.
What if the vaccine boost one part of the immune system but not another? In that case, you might see the vaccine be very effective against symptomatic and asymptomatic Covid, but not more severe disease. If the virus makes its way deep into your lungs, and the lungs are protected by a different immune response that isn’t helped by the vaccine– then, conditional on having gotten to that point, you might not be better off than a non-vaccinated person.
All this to say we should be cautious in putting too much stock in simple mechanistic models.
What if vaccines are all or nothing?
The model behind the claim of extreme vaccine efficacy is that even if your post-vaccine immune response isn’t enough to stop you getting Covid at all, it should be stronger than it would have otherwise been, and you’ll do better at fighting off severe-Covid. This takes vaccine efficacy as a continuous thing.
But maybe the vaccine is 100% effective against all outcomes! So long as it’s correctly transported and administered, that is. Except sometimes vaccines are left at high temperature for too long, the delicate proteins are damaged, and people receiving them are effectively not vaccinated. If this happens 5% of the time, then 95% of people are completely immune to Covid and 5% are identical to not be vaccinated. Whatever chance they had of getting severe Covid before, it’s the same now.
In this world, not knowing whether your vaccine was a dud or not, post-vaccine you should assume you have a 95% reduction across all outcomes equally.
I originally found this argument very persuasive. How could assume that the “continuous” model of vaccine-immune response was true? But actually, most things are continuous in the real world, particularly in biology. People aren’t old or young, but somewhere on a continuous measure. Immune response isn’t an all or nothing affair, and some people’s body’s will produce more antibodies than others. Some enough to stymy any symptoms at all, but some only enough to prevent them getting hospitalized.
One person told me that the mRNA vaccines induce 60x the antibodies of a recovered Covid patient (closest source I found), such that even if your response was weaker, it should still be more than powerful to deal with any actual Covid. Therefore, we should take people still getting quite sick with Covid as a sign that some people’s vaccines must not be working at all.
I would be surprised if complete vaccine failure didn’t occur some of the time, the question is how much. Suppose we have a vaccine that, when it works successfully, confers 100x reduction (99% efficacy). If it fails to work at all 5% of the time, we’d see a
1-(0.01*0.95 + 1*0.05) = 94% efficacy overall. Something like that could be a big part of what we observe.
Also, Kelsey Piper says that she couldn’t find anything about any other vaccines working this way (all or nothing). [Edited: However, as Romeo points out in the comments, Moderna and Pfizer are mRNA vaccines that might not behave like the older types.]
Caveats – read before you act
In my long case for the extreme effectiveness of vaccines, there are some topics of crucial practical importance. These imply that maybe you don’t want to throw caution to the wind just yet. I’m not sure, I didn’t get to looking into these.
Long-Covid
Long-Covid is not an endpoint tracked by any of the studies I’ve looked at. I would think that’d be related to increased viral load and behave like an outcome more severe than just a symptomatic case, but there isn’t data for that. Anecdotally, I’ve heard of a couple of cases where someone experienced mild Covid yet was dramatically affected for months afterwards.
My conservative gut estimate is that your odds or getting long-Covid are reduced by a vaccine by as much as your chance of getting symptomatic Covid at all, but not necessarily any more than that.
Variants
The Israeli study showed supreme efficacy even against the much-feared UK variant (B.1.1.7). However, there are fears the vaccines aren’t not nearly as good against the South African variant (B.1.351) or Brazilian P.1 variant.
A new study published on MedRxiv last week, Evidence for increased breakthrough rates of SARS-CoV-2 variants of concern in BNT162b2 mRNA vaccinated individuals, state that vaccinated individuals were more likely to contract B.1.1.7 and B.1.351 than unvaccinated controls, suggesting both variants are more resistant to vaccination than other strains. (Caveat: I didn’t read the paper in detail.)
. . . we performed a case-control study that examined whether BNT162b2 vaccinees with documented SARS-CoV-2 infection were more likely to become infected with B.1.1.7 or B.1.351 compared with unvaccinated individuals.
Vaccinees infected at least a week after the second dose were disproportionally infected with B.1.351 (odds ratio of 8:1). Those infected between two weeks after the first dose and one week after the second dose, were disproportionally infected by B.1.1.7 (odds ratio of 26:10), suggesting reduced vaccine effectiveness against both VOCs under different dosage/timing conditions.
Not good.
The CDC’s CovidTracker page actually has some nice dashboards for tracking variant proportion, though I haven’t looked into the data quality. Their brief is helpful too.
There’s a breakdown by US state too. I’m in California and am pleased to see that currently, B.1.351 is only 0.3% of the Covid cases and P.1 is 1.6%
Of course, if B.1.351 and friends are resistant to vaccination, we will see them rise in prevalence.
This needs more investigation. Without looking into it more, the sensible strategy would be something like act according to your local prevalences and beliefs about how vaccine-resistant the different strains are. Right now the suspicions are on B.1.351 and P.1, but they’re uncommon in California (0.3% and 1.6% respectively).
If you’ve got the time and skill to look into this more, please do, I can provide you money and glory.
Spreading to the Unvaccinated
Even if the vaccine protects you 200x against more severe outcomes, that doesn’t help the unvaccinated if they catch Covid from you when you had an asymptomatic or mild case. This means that until such time as those you interact with most are vaccinated, you might want to be more conservative in your microCovids.
MicroCOVID.org calculated reductions in contagiousness for vaccinated people (10x reduction for Pfizer/Moderna, 3x for J&J), and that’s what I’d stick to myself if interacting with the unvaccinated. (But hey, California is just about at universal eligibility, now is the time!)
Haven’t finished reading this yet, but in reaction to the opening section… there’s this claim that “One meal at a crowded restaurant is enough to give even a vaccinated person hundreds of microCovids”. Which, regardless of whether it’s true, sounds like it’s probably the wrong way to think about things.
A core part of the microcovid model is that microcovids roughly add. You do a ten-microcovid activity, then a twenty microcovid activity, your total risk is roughly thirty microcovids. Put on a mask, and it cuts microcovids in half.
But with a vaccine, I’d expect the risk to be highly correlated—i.e. how-well-the-vaccine-worked varies from person to person, but it’s fixed for a given person. If that’s true, then eating in a crowded restaurant a hundred times is a lot less than 100x the risk of eating in a crowded restaurant once. My first million not-vaccine-adjusted microcovids are effectively an experiment to find out how well the vaccine worked for me specifically. If it worked well (i.e. I don’t get covid), then the next million exposures are actually much-less-risky.
John and I chatted about this for a bit. We agreed that this is technically true but only of practical significance if you expect to get very large number of microCovids anyway. If you’re someone who can control your exposure, it might not be worth the price of information to find out whether your vaccine worked 100%.
Neat analogy: you’re playing 100-chamber Russian roulette while wearing head armor that is either impervious to bullets or not. If you’re definitely going to pull the trigger 70 times, then you might as well pull more now. You’ll either die or find out you’re impervious either way. If you have a choice of how many pulls, you might still want to keep it to a minimum, say <10, and never be sure whether your armor works or not.
Seems to be of great practical significance to me. If there’s a decent chance that I could return completely to unlimited degrees of interpersonal close contact and have only a 1⁄20 − 1⁄100 chance of getting symptomatic Covid with unlimited amounts of exposure, even if I would get covid many times over while unvaccinated, I’d be quite tempted to do it. If returning to that level of exposure would mean that I’d almost inevitably get Covid eventually, I’d be much more likely to play it safe for at least a few months more and see where things go with infection rates and new variants.
The all-or-nothing vaccine hypothesis is:
If all-or-nothing were true, you would expect the following equality in conditional probability distributions
P(⋅∣infection=true, vaccine=true)=P(⋅∣infection=true, vaccine=false)This is not what we see:
Therefore, all-or-nothing cannot be true.
Am I missing something?
Nope, that seems roughly right. It is I who failed to propagate. Was a cached argument from before I’d looked at the data.
I’ll update the post shortly with this. Thanks for pointing it out.
I now have covid after being vaccinated 3 months ago by Russian Sputnik-V vaccine. For now, it is mild: one day of 38 C, 3 days of 37.5C, only upper level infection, no cough. I lost smell, but it is is slowly returning. Oxygen at my normal level.
Great to hear you only got a mild case.
OT: what’s your impression of COVID spread in Russia? According to official statistics total cumulative deaths and confirmed cases per capita in Russia are maybe half of what they are in Europe and the Americas.
Do you expect this is right, or is there severe underreporting? If no underreporting—did Russia somehow manage the COVID response much better for some reason, or were you guys just lucky?
It is underreporting. There was an analysis of reported deaths, and the picture is grim: It looks like Russia has several times more deaths than was reported and maybe Russia is a world leader. Additional mortality is 338 000 deaths in 2020 (for 140 million population). Almost all my friends had covid.
UPDATE
I added a subsection in the Objections section discussing how the vaccine and control groups in the big Israeli study might be different, and how this should widen confidence intervals.
I think how we relate to this mostly comes down to the properties of the variants, current and future, and whether we trust people to be able to resume tracking microcovids/wearing masks/being cautious again, if it turns out they need to.
I took the CDC strain prevalence data and organized them into a spreadsheet. Note that all of the prevalence-percentages are as of a sample collection period ending March 27, so to get the current prevalence, you have to extrapolate from growth rates. Strain P.1 (informally known as the “Brazil strain”) stands out; this strain made headlines for spreading through a population that had previously had a 75% attack rate, and it has a 2.5x growth rate between the last pair of two-week intervals sampled. Naively extrapolating based on this growth rate, the prevalence in the US will have grown from 1.4% to ~10%, and it will be the dominant strain in less than a month. It is also reported to be more severe.
I have been checking ~daily for data on vaccine effectiveness on this strain. So far there is lab data on this, but no human trial data.
The way these things usually work is, mutation reduces vaccine effectiveness, but only partially. Being fully vaccinated, I think I’m pretty fine with acting normally given the existence of P.1 for now (though I still wear my Versaflo when I visit indoor retail).
What most concerns me is not so much the current variants, as the next ones. Roughly speaking, my model is: The rate of mutation is roughly proportional to the worldwide rate of infections, and the strength and type of selection depends on the vaccines that people had. The worldwide prevalence of COVID is currently very high, and there are a lot of vaccinated and unvaccinated people in close contact with each other. So we’re currently selecting fairly hard for vaccine evasion, but have only started doing so recently. So if vaccine-resistant strains are coming, they will probably come soon. I think we will probably notice reasonably quickly; there’s a lot of sequencing happening, especially in the US.
We also have the recent experience of B.1.351 (the South Africa variant) mostly evading the AstraZeneca vaccine, at least for mild-to-moderate cases. There’s reason to think the mRNA vaccines produce a stronger immune response, which might make them harder to evolve around, and that boosters against future variants will be coming.
Some of the variants have evolved increased severity, but none are reported to have evolved decreased severity. I suspect that there might be a common mechanism, where if the spike protein evolves to be able to enter cells more effectively, this increases both transmissibility and severity together. I’d be interested in someone with more knowledge about virology commenting on this.
Overall, I think fully-vaccinated people in the US currently should not be cowering in their homes, but that it would be a potentially large mistake to disengage and stop paying attention to COVID. The current situation is analogous to one where a snowstorm has just ended, the roads are partially salted, and we’re discussing whether people still need seatbelts. And… yes, obviously? A lot of people went overboard, but the low-hanging fruit of disease prevention is obviously still worth it, and was worth it before the pandemic started. People should still wear masks in public, ventilate their spaces, take vitamin D, and check the news from time to time.
Thanks for this detailed comment. I do think the conclusions of the OP apply for now and one should act on them only so long as a vaccine-evading variant hasn’t become prominent enough to affect overall vaccine protection, and one should be on the lookout for it happening. (I may soon create a mailing list for people to get updates.)
After a few hours of hunting, I ended up finding that GISAID seems to be the central place for getting data on variant data. I couldn’t get access since I don’t have an institutional account, however outbreak.info both has an open-access API and pretty good dashboards for tracking variants.
I’ll have more of a look at them today.
I think now is a bit more like a [potentially brief] Spring and people ought to enjoy the weather before things get frosty again. Though I might update upon looking at the data.
Great post, I noticed I was confused about this the other day.
I think it’s fair for this hypothesis to hang out in the space given that previous vaccines weren’t mRNA based.
Oh yeah, I should have mentioned that mRNA vaccines being a novel type should be factored in. Will edit.
APPENDICES & FOOTNOTES
This comment is a container for comments containing these items.
Appendix B: Breakdown of End Points by Vaccination Status and Time Period for Israel Mass Pfizer Study
Appendix A: Definitions for Israel Nationwide Pfizer Study
As defined by the hospitalizing institution per the Israeli MOH guidelines, consistent with the NIH criteria for severe illness or critical illness:
Individuals who have SpO2 <94% on room air at sea level, a ratio of arterial partial pressure of oxygen to fraction of inspired oxygen (PaO2/FiO2) <300 mm Hg, respiratory frequency >30 breaths/min, or lung infiltrates >50%.
Critical Illness: Individuals who have respiratory failure, septic shock, and/or multiple organ dysfunction.
Updates / Changelog
2021/5/13
I’ve now had a chance to look into variants. Based on: (1) the result from this large study from Qatar, (2) the current prevalences of variants in the US, and (3) the assumption that other variants of concern are no more vaccine-resistant than B.1.351, (4) the current prevalence of variants in the US as per outbreak.info, I conclude that vaccine effective I believe that vaccine effectiveness against getting Covid at all is reduced by a factor of 0.95; however, vaccine-effectiveness against severe disease and death is probably not reduced.
In conclusion, within the United States, I believe that the results of this overall still hold. Huzzah!
2021/4/24
I’ve now spent some time looking into variants and how they might affect vaccine effectiveness. Currently, it seems clear that: (1) certain variants are already pretty widespread and gaining in prevalence in quickly, (2) some of those variants have lab/in- vitro evidence of decreased effectiveness. What is unclear is the real-world clinical significance. I don’t know enough immunology and haven’t read enough to know.
In my current poor epistemic state of ignorance, I would currently guess that vaccines offer somewhat less than 99% effectiveness (for young people against symptomatic), but probably still pretty high, say upwards of 80% or even 90%. And effectiveness against more severe cases is probably still higher. But everything lower confidence than when I wrote this post.
If I can get the chance, I’ll look into this more and provide more updates. For those interested, outbreak.info is an utterly amazing source of data on variants and mutations–both dashboards and collections of relevant papers.
2021/4/22
I removed a short section discussing different false-positive rates among different levels of severity that I now think was confused, following the exchange in this thread.
2021/4/17
I added a subsection in the Objections section discussing how the vaccine and control groups in the big Israeli study might be different, and how this should widen confidence intervals.
I’m pretty happy that this shows the data is consistent with extreme effectiveness in young healthy people, and this post has definitely updated me in that direction. But I’m nervous that the only actual evidence for such high effectiveness is the Israeli observational study, so personally I wouldn’t want to take any actions which depend on being very confident in extreme effectiveness.
I really wish more studies would report on this—seems like information they either already have or could get quite easily.
Apologies in advance for not engaging in detail with the analysis itself—my overall synthesis here is that residual risk does exist post-vaccination and is potentially non-negligible. Personally I’ll be using my Oura ring to detect nighttime temperature spikes and use a high-accuracy at-home test (https://checkit.lucirahealth.com//) if I detect a spike, followed by aggressive treatment with fluvoxamine (+potential antiviral) if the test comes back positive—these safeguards feel sufficient to travel (airplane) to see family, etc without incurring significant risk of long-term health impacts. Appreciate the work to get a tighter bead on the risk itself (and depending on conclusions, some of the above may not have great marginal risk reduction), but wanted to share as one possible mitigation strategy that I expect to be robust even in pessimistic risk-branches.
Where can one get fluvoxamine and antivirals?
Fluvoxamine is a prescription drug in the FDA, so your doctor can prescribe it; https://www.pushhealth.com/ might work as well. Antiviral procurement is similar if the antiviral you’re seeking is a US prescription drug (in any case I’d consider Googling the antiviral name.)
Curated.
This post is the product of weeks of research, discussion, and waiting to see if counter-evidence showed up. Folk on the LW team were more hesitant than usual to curate this post (normally, curation does not mark a post as having a particularly strong epistemic status). But after several weeks, we’re confident enough in this post to signal boost it. I hope it gives people information that helps them to decide how to act in the post-vaccine world.
What of the 6x or worse effectiveness against a few strains gaining currency e.g. the brazilian one? Seems still valuable under this model.
I think we need to track data about these variants and their prevalence.
I think this should be “99% effective”.
Thanks, fixed.
Related: https://marginalrevolution.com/marginalrevolution/2021/05/cold-storage-no-longer-a-constraint.html suggests that:
cold-chain requirements had more margin for error than any of us had thought
the process that produced excess margin for error in this case likely produced excess margin for error in other relevant areas
(tentative) Should this make less plausible a line of reasoning that goes through “Except sometimes vaccines are left at high temperature for too long, the delicate proteins are damaged, and people receiving them are effectively not vaccinated...”? I’m not sure yet, and I don’t know how central this particular line is to the overall argument.
That makes sense, and now you mention it, I heard the same about injectable monoclonal antibodies. At first they thought a few hours at room temperature would destroy them, turned out that’s probably false.
> Of course, the outcomes we’re interested in are hospitalization, severe Covid, and death. I’d expect the false positives on these to be lower than for having Covid at all, but across tens of thousands of people (the Israel study did still have thousands even in later periods), it’s not crazy that some people would be very ill with pneumonia and also get a false positive on Covid.
Does this observation undermine the claim of a general trend in effectiveness with increasing severity of disease? That is, if false positives bias the measured effectiveness downward, and if false positives are more frequent with less severe severe, then the upward trend is less robust and our use of it to extrapolate into places where the error bars are naively large is less convincing.
I don’t think it undermines it. What matters is the relative frequency of true cases [1] vs false positives.
With less severe disease (e.g. symptomatic), we might have a frequency of 1% true cases in the population, plus 0.1% false-positive rate. The true cases greatly outnumber the false-positives.
In contrast, vaccinated death from Covid might be only 0.001% in the population, while false-positive deaths are 0.01%. Here the false-positives dominate.
So even though the absolute false-positive rate is lower in more severe cases (because it’s harder to misattribute deaths than get wrong test results), it still dominates the effectiveness results more because it’s larger than the rate of actual occurrences of the event.
[1] I say “true cases” deliberately instead of true-positives, because I mean to say the objective underlying frequency of the event, not true-positive detection rate.
You’ve given some toy numbers as a demonstration that the claim needn’t necessarily be undermined, but the question is whether it’s undermined by the actual numbers.
I thought about this for a while, and I think the entailment you point out is correct and we can’t be sure the numbers turn out as in my example.
But also, I think I got myself confused when writing the originally cited passage. I was thinking about how there will be a smaller absolute number of false-positive deaths than the absolute number of false-positive symptomatic cases, because there are fewer death generally. That doesn’t require the false-positive rates to be different to be true.
Also thinking about it, the mechanisms by which the false-positive rate would be lower on severe outcomes that I’d been thinking of don’t obviously hold. It’s probably more like if someone had a false-positive test and then had pneumonia symptoms, it’d be mistaken for Covid, and the rate of that happening is only dependent on the regular Covid test false-positive rate.
While I’m generally a person who considers low bureaucracy very important, I think it’s very sad that we didn’t manage to run a single vaccine trials with endpoints for long-covid.
Besides telling us about how effective vaccines are against long-covid it would have also told us about how big of a deal long covid is in general because we would have a controlled setting for studying long covid.
This might have been one of the biggest mistakes of 2020.
Antibody count is not the only thing that matters. Antibody diversity also matters. A recovered Covid patient has antibodies against multiple Covid proteins while a mRNA vaccinated person only has antibodies against the spike protein.
That means that if you are exposed to a variant with mutated spike protein you are better of as a recovered Covid patient because you still have antibodies against other proteins.
“I wrote first wrote”
Thanks for the post!
Thanks for mentioning the typo!
Very nicely researched and explained analysis. Contains several points that I had not thought about much before if at all, including the analysis of false negatives in this particular context.
I’d be interested to hear more thoughts on your mechanistic explanation from commenters with more expertise in immunology.
Also interested to hear how you and other commenters are planning to change your own Covid risk-taking behavior after being vaccinated. I am coming up on two weeks after my second Pfizer shot and debating that for myself.
It seems the caveat regarding selection effects, that vaccinated people might be prone to more health-seeking behavior and might risk compensate, could swamp many of the other careful arguments. Is there any way we could get a rough estimate on these selection effects?
The variants caveat is scary, but at least we can monitor variants over time and try to adjust our risk behavior accordingly.