I don’t think I entirely agree with your characterization of the situation and would like to push back a bit.
> 2. It is illegal to continue a trial to study the drug, because it has been proven so safe and effective that it isn’t ethical to not give the drug to half the patients.
Sorry, but where are you getting this from? I am pretty sure it is not illegal to continue a trial if it crosses an efficacy or futility boundary in the interim (happy to be corrected on this point). In fact, when futility boundaries are crossed in interim analyses (ie, when it seems a drug is most likely to be ineffective), the trials are often continued after discussion by the data monitoring committee. There are specific ways to handle this—statistically this means setting up a “non-binding” interim analysis boundary. In other words, you can choose to continue the trial even if the drug is deemed “too ineffective” (or “too effective) in an interim analysis.
The reason trials rarely (never?) continue after crossing an efficacy boundary in the interim is because there’s no incentive for the sponsor (ie, the drug company) to continue once a drug has been deemed effective. And to recruit more patients costs time and money for essentially no added benefit. You can choose to read this uncharitably (Big Pharma is skimping on money) or charitably (they are shortening time to approval and helping more people earlier).
3. Who, if they weren’t in the study, couldn’t get the drug at all, because it is illegal due to not being proven safe and effective yet.
4. So now no one gets added to the trial so those who would have been definitely don’t get Paxlovid, and are several times more likely to die.
As I see it, the two possible scenarios are this: 1) You stop a trial early and submit data for FDA approval. This gets the drug in the hands of the masses earlier. 2) You run the full trial and then submit data for approval. Wider access to the drug is delayed, costing lives.
The first option is better because I think we can reasonably assume that the time it takes for FDA to approve the drug is same in both of the cases (and so equal no. of people will be harmed while waiting for the FDA to approve the drug). However, wider access to the drug is established quicker in the first scenario, thus saving more lives over the long run.
6. For the rest of time we will now hear about how it was only seven deaths and we can’t be sure Paxlovid works or how well it works, and I expect to spend hours arguing over exactly how much it works.
7. For the rest of time people will argue the study wasn’t big enough so we don’t know the Paxlovid is safe.
8. Those arguments will then be used both by people arguing to not take Paxlovid, and people who want to require other interventions because of these concerns.
IMO, this makes your critique kind of incoherent to me. Should the FDA be more conservative in this scenario or not? If you wait for more precise treatment estimates by running a full trial, then there’s a delay which costs lives (as I outlined previously). Or you could be less conservative given the circumstances, make do with the best statistical methods you’ve got and accept results from an interim analysis.
FWIW, the statistical literature on stopping trials early is indeed mixed (eg, you’re prone to over-estimating effect sizes although there are ways to adjust for it now but it’s an ongoing area of research and so on). However results from interim analysis are deemed perfectly acceptable by drug regulatory authorities and ideally, stopping early should not be inflating Type-I error etc. (statistical methods try to take this into account) So, results from an interim analysis should be as good as running a “full” clinical trial.
I think most of us agree that FDA should be less conservative (especially during covid) and that’s exactly what it’s doing by accepting results from interim analyses. This is actually one of the few instances where FDA’s decision making is anti-conservative. And again, like I said before, it is not illegal to run full trials if you get positive results from interim analyses.
So, to summarize, I am kind of confused by your characterization of the situation. I think FDA should definitely be doing better in emergency situations but the specifics of your criticism here don’t seem convincing to me. What is the proposed alternative here? Don’t accept interim analyses? Force drug companies to continue the trial if interim analyses are beneficial while waiting for drug approval?
I don’t think I entirely agree with your characterization of the situation and would like to push back a bit.
> 2. It is illegal to continue a trial to study the drug, because it has been proven so safe and effective that it isn’t ethical to not give the drug to half the patients.
Sorry, but where are you getting this from? I am pretty sure it is not illegal to continue a trial if it crosses an efficacy or futility boundary in the interim (happy to be corrected on this point). In fact, when futility boundaries are crossed in interim analyses (ie, when it seems a drug is most likely to be ineffective), the trials are often continued after discussion by the data monitoring committee. There are specific ways to handle this—statistically this means setting up a “non-binding” interim analysis boundary. In other words, you can choose to continue the trial even if the drug is deemed “too ineffective” (or “too effective) in an interim analysis.
The reason trials rarely (never?) continue after crossing an efficacy boundary in the interim is because there’s no incentive for the sponsor (ie, the drug company) to continue once a drug has been deemed effective. And to recruit more patients costs time and money for essentially no added benefit. You can choose to read this uncharitably (Big Pharma is skimping on money) or charitably (they are shortening time to approval and helping more people earlier).
As I see it, the two possible scenarios are this: 1) You stop a trial early and submit data for FDA approval. This gets the drug in the hands of the masses earlier. 2) You run the full trial and then submit data for approval. Wider access to the drug is delayed, costing lives.
The first option is better because I think we can reasonably assume that the time it takes for FDA to approve the drug is same in both of the cases (and so equal no. of people will be harmed while waiting for the FDA to approve the drug). However, wider access to the drug is established quicker in the first scenario, thus saving more lives over the long run.
IMO, this makes your critique kind of incoherent to me. Should the FDA be more conservative in this scenario or not? If you wait for more precise treatment estimates by running a full trial, then there’s a delay which costs lives (as I outlined previously). Or you could be less conservative given the circumstances, make do with the best statistical methods you’ve got and accept results from an interim analysis.
FWIW, the statistical literature on stopping trials early is indeed mixed (eg, you’re prone to over-estimating effect sizes although there are ways to adjust for it now but it’s an ongoing area of research and so on). However results from interim analysis are deemed perfectly acceptable by drug regulatory authorities and ideally, stopping early should not be inflating Type-I error etc. (statistical methods try to take this into account) So, results from an interim analysis should be as good as running a “full” clinical trial.
I think most of us agree that FDA should be less conservative (especially during covid) and that’s exactly what it’s doing by accepting results from interim analyses. This is actually one of the few instances where FDA’s decision making is anti-conservative. And again, like I said before, it is not illegal to run full trials if you get positive results from interim analyses.
So, to summarize, I am kind of confused by your characterization of the situation. I think FDA should definitely be doing better in emergency situations but the specifics of your criticism here don’t seem convincing to me. What is the proposed alternative here? Don’t accept interim analyses? Force drug companies to continue the trial if interim analyses are beneficial while waiting for drug approval?