I dug into the Israel vaccine data some. Full data is lacking and I strongly suspected the true VE is significantly higher, based on the UK’s 78.2-93.2% estimate for Pfizer 2 dose. Below is my thought process.
TL;DR I thought I would find a clear reason the Israeli data was wrong. I tried to see if the interval was so large that the Israeli estimate was meaningless or if there was a huge bias, but nothing solid came up, so I’ve gone from “confident” to “somewhat nervous”.
It looks like for the recent numbers they took each age group and did a VE estimate, and combined the result. They do an example point estimate of VE for age 35-44 and get 55.7% efficacy, based on 47 vaccinated and 15 unvaccinated infections in a population that was 7.08:1 [2nd dose 7+ days ago]:[No vaccine]. Population is in units of person*days.
That’s not many cases (for this age bin). What are the bounds on the example efficacy? Turns out there’s a Bayesian way to calculate this, which I won’t write out. Assuming I did it right, the 95% credibility interval is 16.5-74.3% for this age group.
So could the 64% expected value of VE be similarly low-confidence? There’s no obvious way to guess the brackets on the 64% for the full population without knowing the relative population sizes for all groups. But it looks like the total population count is 257:1271, which is something like 25x the data points. I expect a tighter interval, but not necessarily 5x tighter because of complicated statistics.
So long story short, I’m leaning towards the UK data being correct. But my expectations were that the Israeli data would be confidently falsified with a couple hours of thought, but this didn’t happen, so I’m no longer as highly confident.
I dug into the Israel vaccine data some. Full data is lacking and I strongly suspected the true VE is significantly higher, based on the UK’s 78.2-93.2% estimate for Pfizer 2 dose. Below is my thought process.
TL;DR I thought I would find a clear reason the Israeli data was wrong. I tried to see if the interval was so large that the Israeli estimate was meaningless or if there was a huge bias, but nothing solid came up, so I’ve gone from “confident” to “somewhat nervous”.
Here’s the announcement: https://www.gov.il/en/departments/news/06072021-04
And a more quantitative version (fortunately Google Translate was pretty good at Modern Hebrew, at least in this context): https://www.gov.il/BlobFolder/news/06072021-04/en/NEWS_Corona_vaccine-eficacy.pdf
They say they used the same methodology for Delta effectiveness as what’s in this older paper: https://www.gov.il/BlobFolder/news/06052021-02/ru/NEWS_Corona_lancet-article.pdf
It looks like for the recent numbers they took each age group and did a VE estimate, and combined the result. They do an example point estimate of VE for age 35-44 and get 55.7% efficacy, based on 47 vaccinated and 15 unvaccinated infections in a population that was 7.08:1 [2nd dose 7+ days ago]:[No vaccine]. Population is in units of person*days.
That’s not many cases (for this age bin). What are the bounds on the example efficacy? Turns out there’s a Bayesian way to calculate this, which I won’t write out. Assuming I did it right, the 95% credibility interval is 16.5-74.3% for this age group.
So could the 64% expected value of VE be similarly low-confidence? There’s no obvious way to guess the brackets on the 64% for the full population without knowing the relative population sizes for all groups. But it looks like the total population count is 257:1271, which is something like 25x the data points. I expect a tighter interval, but not necessarily 5x tighter because of complicated statistics.
My other thought is that there is a bias. Something that seems pretty funky is the usage of person-days. During the interval June 6-July 3, the first two weeks had <10% as many cases as the last two weeks. https://ourworldindata.org/explorers/coronavirus-data-explorer?zoomToSelection=true&time=2021-06-06..2021-07-03&pickerSort=asc&pickerMetric=location&Metric=Confirmed+cases&Interval=New+per+day&Relative+to+Population=false&Align+outbreaks=false&country=~ISR Since people strictly leave the unvaccinated group and strictly enter the fully vaccinated group, the average fully vaccinated person-day was on a more case-heavy day than the average unvaccinated person-day. Combined with 1 week being too short to account for the actual effect of dose 2 on positives, maybe this introduces a heavy bias? The paper referenced for methodology includes numbers adjusted for week, but it’s not clear if that means week-of-vaccination or weekly cases, and it’s not clear if the Delta numbers were adjusted this way. So seems reasonable.
But only 1.97% of the population was vaccinated in this interval, and only 0.29% got Dose 2 in [interval − 1 week]. https://ourworldindata.org/explorers/coronavirus-data-explorer?zoomToSelection=true&time=2021-06-06..2021-07-03&pickerSort=asc&pickerMetric=location&Metric=People+vaccinated&Interval=New+per+day&Relative+to+Population=true&Align+outbreaks=false&country=~ISR and https://ourworldindata.org/explorers/coronavirus-data-explorer?zoomToSelection=true&time=2021-05-30..2021-06-26&pickerSort=asc&pickerMetric=location&Metric=People+fully+vaccinated&Interval=7-day+rolling+average&Relative+to+Population=true&Align+outbreaks=false&country=~ISR Compared to the June 6 numbers, the unvaccinated population count only had a ~5.3% decrease by the end, and the fully vaccinated had a 0.5% relative increase by the end. The only way I could see this making a big difference is if most cases and new vaccinations were in the same age bin (since the bin would have > 5.3%/.5% relative changes, and more cases mean more weight). It would not be surprising if this was a big enough factor to account for differences with UK data, given the age distribution of cases.
So long story short, I’m leaning towards the UK data being correct. But my expectations were that the Israeli data would be confidently falsified with a couple hours of thought, but this didn’t happen, so I’m no longer as highly confident.