Well, this post was just crying out for some embedded predictions! So here we go:
The radvac vaccine will have serious side effects (i.e. besides stuffy nose for a day) for >50% of people who try itThe radvac vaccine will have serious side effects for >1% of people who try it The radvac vaccine works. That is, it immunizes against COVID, in >50% of people, using the dosage in the paper with 2 booster shotsThe radvac vaccine works in a limited fashion. That is, it immunises against COVID infection via the noise only, in >50% of people, using the dosage in the paper with 2 booster shots For a given person who administers radvac using the dosage in the paper and 2 booster shots, how likely are they to be immunised against COVID?The radvac vaccine induces antibodies detectable in a standard commercial blood test in most people, using the dosage in the paper with 2 booster shotsThe radvac vaccine induces antibodies detectable in a standard commercial blood test in most people, using "more dakka", for some reasonable version of "more dakka"
Thanks johnswentworth for help with some of the operationalisations!
I included many different ones, as I think it is often good try to triangulate high stakes questions via different operationalisations. This reduces some some “edge-case noise” stemming from answering vague questions in overly specific ways.
First, base rates are critical. Looking at potential drugs overall, the rate of approvals due to safety alone—i.e. “Investigational New Drugs” to phase-II efficacy trials, is very low. Phase 1 trials are typically 80-100 people, and most don’t manage to make it past that stage. It would take much stronger evidence than I have seen to think that this vaccine is going to be outside of the norm.
Second, even if the process as done was safe, I can’t imagine that greater than 99% of people manage to do this without screwing up in some serious way. That’s less true of the LW crowd, but I don’t think people are aware of how dumb the mistakes that get made are, or how much quality control matters, and how difficult it is with trying to enforce it for DIY projects.
Lastly, I’m well within the consensus for almost all the rest of the questions—I think it probably works in most cases, and I think it will have side effects in far fewer than 50% of cases.
(But another place I’m a bit outside the consensus is that I think it’s unlikely to trigger standard antibody tests, since standard antibody tests are looking for antibodies against a specific part of the virus, and I’m unsure, reading the “Antibodies and B-cell immune response” section of the white paper, that standard tests would detect the elicited types of NABs.)
As someone considering taking it, I’d be interested in whether you have models of particular side effects or severities that might happen and why? Do you just have vague “accidents and harmful unknown unknowns aren’t that unlikely here”?
I have not currently read the paper or looked into anything very hard. But one question I’d have is “given the ingredients you’re working with, is there are particularly obvious way to mix these on purpose that’d result in something harmful happening?”
Mostly vague “accidents and harmful unknown unknowns aren’t that unlikely here”—because we have data on baseline success at “not have harmful side effects,” and it is low. We also know that lots of important side effects are unusual, so the expected loss can be high even after a number of “successes,” and this is doubly true because no-one is actually tracking side effects. We don’t know much about efficacy either, but again, on base rates it is somewhat low. (Base rates for mRNA are less clear, and may be far higher—but these sequences are unfiltered, so I’m not sure even those bse rates would apply.)
Finally, getting the adjuvants to work is typically tricky for vaccines, and I’d be very concerned about making them useless, or inducing reactions to something other than the virus. But if you want to know about intentional misuse, it’s relatively low. I would wonder about peanut protein to induce you to develop a new allergy because you primed your immune system to react to a new substance, but you’d need someone more expert than I.
Overall, I’d be really happy taking bets that in 20 years, looking back with (hopefully) much greater understanding of mRNA vaccines, a majority of immunologists would respond to hearing details about this idea with a solid “that’s idiotic, what the hell were those idiots thinking?” (If anyone wants to arrange details of this bet, let me know—it sounds like a great way to diversify and boost my expected retirement returns.)
> The radvac vaccine will have serious side effects (i.e. besides stuffy nose for a day) for >50% of people who try it
It should be well below 1%. Firstly, if it were that bad as to cause serious side effects for >50% of people who try it, would the RaDVaC team risk promoting it? Secondly, if it were that bad, wouldn’t we hear bad stories about side effects? Thirdly, getting serious side effects accidentally in >50% cases sounds pretty hard on its own.
> The radvac vaccine induces antibodies detectable in a standard commercial blood test in most people, using the dosage in the paper with 2 booster shots
<1%, because RaDVaC team has tried it and didn’t manage to get any positive result.
> The radvac vaccine induces antibodies detectable in a standard commercial blood test in most people, using “more dakka”, for some reasonable version of “more dakka”
This greatly depends on what “more dakka” and “reasonable version” means. I assume that “reasonable version” implies “doesn’t cause too much harm due to immune system overstimulation”. If “more dakka” means simply a higher dosage, then I think, that this is unlikely (5%), because 1) RaDVaC team experimented on themselves quite a bit, they received a lot of dakka, but no commercial blood test detection, 2) RaDVaC team seems reasonable enough to try this approach if it looked promising. If “more dakka“ includes stronger adjuvants (chitosan is considered a weak, but safe one), then it is much more likely (20%?), because RaDVaC team didn’t investigate those (for a reason) and it sounds plausible that you can get an immune response by irritating the immune system really, really strongly.
I would prefer “immunizes against COVID-19” to be better defined when asking the question. Immunizing against becoming systomatic with COVID-19 is a different value then immunizing against not being infectious with COVID-19 and both of those matter. Preventing rate of hospitalization and rate of death would also be important.
In epidemiology / medicine, etc. “Immunizes” has a technical meaning—it means you cannot contract or carry the disease. (i.e. not that you don’t get symptoms.)
By that definition Moderna and BionTech don’t have vaccines that are proven to immunize people against COVID-19. That technical meaning might be used in some communities but I think if you would ask the median rationalist whether or not there was a clinical trial whether the Moderna or Pfizer vaccine immunizes people against COVID-19 they would say “yes, there was such a trial”.
If a journalist would ask someone at the FDA or CDC the question whether such a trial I doubt they would get the answer “technically there wasn’t a trial that showed the those vaccines immunize anyone”.
The idea that the technical meaning of a scientific term is something that isn’t operationalized seems to me also problematic. There might be people in epidemiology who believe that but it’s ontologically problematic and causes a lot of harm.
“Immunity” and “efficacy” seem like they should refer to the same thing, but they really don’t. And if you talk to people at the FDA, or CDC, they should, and probably would, talk about efficacy, not immunity, when talking about these vaccines.
And I understand that the technical terms and usage aren’t the same as what people understand, and I was trying to point out that for technical usage, the terms don’t quite mean the things you were assuming.
And yes, the vaccines have not been proven to provide immunizing protection—which again, is different than efficacy. (But the vaccines do almost certainly provide immunizing protection for some people, just based on the obvious prior information and the current data—though it’s unclear how well they do so, at how long after the vaccine.)
And, to make things worse, even efficacy is unclearly defined. It gets defined in each clinical trial - differently for each drug/vaccine/etc. and I don’t think it actually mean the same thing for the currently approved COVID-19 vaccines. It’s pretty similar, stopping symptomatic cases, but even given the same endpoint, it’s not necessarily identical, since the studies picked how to measure the endpoints independently, and differently.
Well, this post was just crying out for some embedded predictions! So here we go:
Thanks johnswentworth for help with some of the operationalisations!
I included many different ones, as I think it is often good try to triangulate high stakes questions via different operationalisations. This reduces some some “edge-case noise” stemming from answering vague questions in overly specific ways.
@Davidmanheim you’re a pretty big outlier here, and this is also the kind of question where I’d trust your judgement a fair bit:
So curious if you wanted to elaborate a bit on your model?
First, base rates are critical. Looking at potential drugs overall, the rate of approvals due to safety alone—i.e. “Investigational New Drugs” to phase-II efficacy trials, is very low. Phase 1 trials are typically 80-100 people, and most don’t manage to make it past that stage. It would take much stronger evidence than I have seen to think that this vaccine is going to be outside of the norm.
Second, even if the process as done was safe, I can’t imagine that greater than 99% of people manage to do this without screwing up in some serious way. That’s less true of the LW crowd, but I don’t think people are aware of how dumb the mistakes that get made are, or how much quality control matters, and how difficult it is with trying to enforce it for DIY projects.
Lastly, I’m well within the consensus for almost all the rest of the questions—I think it probably works in most cases, and I think it will have side effects in far fewer than 50% of cases.
(But another place I’m a bit outside the consensus is that I think it’s unlikely to trigger standard antibody tests, since standard antibody tests are looking for antibodies against a specific part of the virus, and I’m unsure, reading the “Antibodies and B-cell immune response” section of the white paper, that standard tests would detect the elicited types of NABs.)
As someone considering taking it, I’d be interested in whether you have models of particular side effects or severities that might happen and why? Do you just have vague “accidents and harmful unknown unknowns aren’t that unlikely here”?
I have not currently read the paper or looked into anything very hard. But one question I’d have is “given the ingredients you’re working with, is there are particularly obvious way to mix these on purpose that’d result in something harmful happening?”
Mostly vague “accidents and harmful unknown unknowns aren’t that unlikely here”—because we have data on baseline success at “not have harmful side effects,” and it is low. We also know that lots of important side effects are unusual, so the expected loss can be high even after a number of “successes,” and this is doubly true because no-one is actually tracking side effects. We don’t know much about efficacy either, but again, on base rates it is somewhat low. (Base rates for mRNA are less clear, and may be far higher—but these sequences are unfiltered, so I’m not sure even those bse rates would apply.)
Finally, getting the adjuvants to work is typically tricky for vaccines, and I’d be very concerned about making them useless, or inducing reactions to something other than the virus. But if you want to know about intentional misuse, it’s relatively low. I would wonder about peanut protein to induce you to develop a new allergy because you primed your immune system to react to a new substance, but you’d need someone more expert than I.
Overall, I’d be really happy taking bets that in 20 years, looking back with (hopefully) much greater understanding of mRNA vaccines, a majority of immunologists would respond to hearing details about this idea with a solid “that’s idiotic, what the hell were those idiots thinking?” (If anyone wants to arrange details of this bet, let me know—it sounds like a great way to diversify and boost my expected retirement returns.)
This was great.
> The radvac vaccine will have serious side effects (i.e. besides stuffy nose for a day) for >50% of people who try it
It should be well below 1%. Firstly, if it were that bad as to cause serious side effects for >50% of people who try it, would the RaDVaC team risk promoting it? Secondly, if it were that bad, wouldn’t we hear bad stories about side effects? Thirdly, getting serious side effects accidentally in >50% cases sounds pretty hard on its own.
> The radvac vaccine induces antibodies detectable in a standard commercial blood test in most people, using the dosage in the paper with 2 booster shots
<1%, because RaDVaC team has tried it and didn’t manage to get any positive result.
> The radvac vaccine induces antibodies detectable in a standard commercial blood test in most people, using “more dakka”, for some reasonable version of “more dakka”
This greatly depends on what “more dakka” and “reasonable version” means. I assume that “reasonable version” implies “doesn’t cause too much harm due to immune system overstimulation”. If “more dakka” means simply a higher dosage, then I think, that this is unlikely (5%), because 1) RaDVaC team experimented on themselves quite a bit, they received a lot of dakka, but no commercial blood test detection, 2) RaDVaC team seems reasonable enough to try this approach if it looked promising. If “more dakka“ includes stronger adjuvants (chitosan is considered a weak, but safe one), then it is much more likely (20%?), because RaDVaC team didn’t investigate those (for a reason) and it sounds plausible that you can get an immune response by irritating the immune system really, really strongly.
That’s false, they got several positive anitbody results in ~June or so last year. See a comment elsewhere on this post.
I think he might mean commercial blood antibody tests specifically?
Yes, exactly. “None of us has tested positive using insensitive commercial point-of-care tests”
I would prefer “immunizes against COVID-19” to be better defined when asking the question. Immunizing against becoming systomatic with COVID-19 is a different value then immunizing against not being infectious with COVID-19 and both of those matter. Preventing rate of hospitalization and rate of death would also be important.
In epidemiology / medicine, etc. “Immunizes” has a technical meaning—it means you cannot contract or carry the disease. (i.e. not that you don’t get symptoms.)
By that definition Moderna and BionTech don’t have vaccines that are proven to immunize people against COVID-19. That technical meaning might be used in some communities but I think if you would ask the median rationalist whether or not there was a clinical trial whether the Moderna or Pfizer vaccine immunizes people against COVID-19 they would say “yes, there was such a trial”.
If a journalist would ask someone at the FDA or CDC the question whether such a trial I doubt they would get the answer “technically there wasn’t a trial that showed the those vaccines immunize anyone”.
The idea that the technical meaning of a scientific term is something that isn’t operationalized seems to me also problematic. There might be people in epidemiology who believe that but it’s ontologically problematic and causes a lot of harm.
“Immunity” and “efficacy” seem like they should refer to the same thing, but they really don’t. And if you talk to people at the FDA, or CDC, they should, and probably would, talk about efficacy, not immunity, when talking about these vaccines.
And I understand that the technical terms and usage aren’t the same as what people understand, and I was trying to point out that for technical usage, the terms don’t quite mean the things you were assuming.
And yes, the vaccines have not been proven to provide immunizing protection—which again, is different than efficacy. (But the vaccines do almost certainly provide immunizing protection for some people, just based on the obvious prior information and the current data—though it’s unclear how well they do so, at how long after the vaccine.)
And, to make things worse, even efficacy is unclearly defined. It gets defined in each clinical trial - differently for each drug/vaccine/etc. and I don’t think it actually mean the same thing for the currently approved COVID-19 vaccines. It’s pretty similar, stopping symptomatic cases, but even given the same endpoint, it’s not necessarily identical, since the studies picked how to measure the endpoints independently, and differently.
I didn’t mean to predict on this; I was just trying to see number of predictions on first one. Turns out that causes prediction on mobile
You can un-predict by clicking on the prediction block again.