First I’d like to apologize for not reaching out to you before publishing my critique, I tried to integrate your responses from our email conversation but should have given you a chance to respond before publishing.
A minor point, for the data extrapolation you are reading the graph incorrectly. A higher growth difference meant that the growth rate (a rough approximation of R0) fell more sharply. The point of this section was not that the effect wasn’t large, but that it pointed weakly in the wrong direction. Regions which increased mask wearing the most had their growth rates fall more slowly. I don’t think this is strong evidence, but it does point against the effectiveness of mask wearing.
This is the spreadsheet used to compute the graph:
The core point of dispute if I understand it correctly is that knowing the absolute level of mask wearing in a region does not give evidence as to the overall R0 (even taking into account mobility and NPIs), but knowing the change in mask wearing over time gives evidence as to the change in R0.
In the model in your paper the absolute effect and relative effect are not disambiguated and an effect size mean of 25% reduction is observed.
In your proposed specification you try to isolate the relative effect by zeroing out starting mask wearing and observe a higher impact.
In my proposed specification I try to isolate the absolute effect by zeroing out the relative changes and observe a smaller impact.
These two observations don’t contradict each other. The data is consistent with two distinct causal stories.
High inherent transmission → Mask Wearing → Lower transmission. This is your preferred model and indicates that the absolute effect is spurious.
High inherent transmission → Mask Wearing + Lower Transmission later. This is my preferred model and indicates that the relative effect is spurious.
One reason to favor the second story is that although the model is described as measuring R_0, it is actually measuring Rt. The model does not include population infections as a modifier to the growth rate. This is a known causal factor which would artificially make masks look more effective.
As further evidence that our actual belief states are not too far apart, 15-25% reduction in case load (different from R0) was my best guess for the results of the mask RCT. I will make the $100 bet with the caveat being that payment is in the form of a donation to GiveWell.
Great! Comment below if you like this wording and this can be our bond:
”Gavin bets 100 USD to GiveWell, to Mike’s 100 USD to GiveWell that the results of NCT04630054 will show a median reduction in Rt > 15.0 % for the effect of a whole population wearing masks [in whatever venues the trial chose to study].”
I can’t accept the wording because the masking study is not directly measuring Rt. I would prefer this wording
“Gavin bets 100 USD to GiveWell, to Mike’s 100 USD to GiveWell that the results of NCT04630054 will show a median reduction in cumulative cases > 15.0 % for the effect of a whole population wearing masks [in whatever venues the trial chose to study].”
For the cloth mask they got a 5% reduction in seroprevalence (equivalent to 15% for 100% increase) and for surgical masks they got an 9.3% reduction (equivalent to 28% for 100% increase).
I unequivocally lost the bet and will send my donation. Let me know if you have a preferred charity.
I’d like to respond to some of the points raised.
First I’d like to apologize for not reaching out to you before publishing my critique, I tried to integrate your responses from our email conversation but should have given you a chance to respond before publishing.
A minor point, for the data extrapolation you are reading the graph incorrectly. A higher growth difference meant that the growth rate (a rough approximation of R0) fell more sharply. The point of this section was not that the effect wasn’t large, but that it pointed weakly in the wrong direction. Regions which increased mask wearing the most had their growth rates fall more slowly. I don’t think this is strong evidence, but it does point against the effectiveness of mask wearing. This is the spreadsheet used to compute the graph:
The core point of dispute if I understand it correctly is that knowing the absolute level of mask wearing in a region does not give evidence as to the overall R0 (even taking into account mobility and NPIs), but knowing the change in mask wearing over time gives evidence as to the change in R0.
In the model in your paper the absolute effect and relative effect are not disambiguated and an effect size mean of 25% reduction is observed.
In your proposed specification you try to isolate the relative effect by zeroing out starting mask wearing and observe a higher impact.
In my proposed specification I try to isolate the absolute effect by zeroing out the relative changes and observe a smaller impact.
These two observations don’t contradict each other. The data is consistent with two distinct causal stories.
High inherent transmission → Mask Wearing → Lower transmission. This is your preferred model and indicates that the absolute effect is spurious.
High inherent transmission → Mask Wearing + Lower Transmission later. This is my preferred model and indicates that the relative effect is spurious.
One reason to favor the second story is that although the model is described as measuring R_0, it is actually measuring Rt. The model does not include population infections as a modifier to the growth rate. This is a known causal factor which would artificially make masks look more effective.
As further evidence that our actual belief states are not too far apart, 15-25% reduction in case load (different from R0) was my best guess for the results of the mask RCT. I will make the $100 bet with the caveat being that payment is in the form of a donation to GiveWell.
Great! Comment below if you like this wording and this can be our bond:
”Gavin bets 100 USD to GiveWell, to Mike’s 100 USD to GiveWell that the results of NCT04630054 will show a median reduction in Rt > 15.0 % for the effect of a whole population wearing masks [in whatever venues the trial chose to study].”
I can’t accept the wording because the masking study is not directly measuring Rt. I would prefer this wording
“Gavin bets 100 USD to GiveWell, to Mike’s 100 USD to GiveWell that the results of NCT04630054 will show a median reduction in cumulative cases > 15.0 % for the effect of a whole population wearing masks [in whatever venues the trial chose to study].”
Sounds fine. Just noticed they have a cloth and a surgical treatment. Take the mean?
Sure.
My current belief state is that cloth masks will reduce case load by ~15% and surgical masks by ~20%.
Without altering the bet I’m curious as to what your belief state is.
The study was just released https://www.poverty-action.org/study/impact-mask-distribution-and-promotion-mask-uptake-and-covid-19-bangladesh
For the cloth mask they got a 5% reduction in seroprevalence (equivalent to 15% for 100% increase) and for surgical masks they got an 9.3% reduction (equivalent to 28% for 100% increase).
I unequivocally lost the bet and will send my donation. Let me know if you have a preferred charity.
Givewell’s fine!
Thanks again for caring about this.