I’ve now run 9 ELISA tests. The main result is noise: negative controls are all over the map, sometimes very blue (i.e. positive), sometimes not blue at all. I did see more positive results in the experimental group than I’d expect from noise alone, but I haven’t gotten the noise to a point where results are consistently reproducible.
Meanwhile, I also ran one very simple test: I snorted a batch of the peptides, without the chitosan or anything else—just peptides in deionized water. Previously, on doses 3-6 of the vaccine, I had consistently been congested for a couple days after (and not congested the rest of the week), which strongly indicates an immune response. However, that response could have been to the chitosan or other contents of the vaccine, rather than the peptides. This test put that possibility to rest: after snorting just the peptides, I was very obviously congested for a couple days, in basically the same way as after the vaccine doses.
So thanks to that simple test, I personally am now pretty highly confident that I have an immune response to these peptides. Unfortunately it’s not as legible as an ELISA test, so you should not necessarily be quite as convinced by this.
Now, this still leaves the question of whether an immune response to these peptides translates into an immune response to COVID. It could be that e.g. the conformation of the peptides’ corresponding sequence within full COVID proteins is different enough that it doesn’t carry over. Personally, though, I consider this a much less likely failure mode, for two reasons. First, the white paper indicates that the peptides were chosen based on antibodies developed by people who actually had COVID. Second, whether antibodies against these peptides bind the real proteins is something which I would not expect to vary much from person to person, so if it’s worked for a few people it should work for everyone—and the whitepaper does indicate that multiple groups have seen positive results testing for binding against the full proteins.
None of this puts my confidence up close to 99%, but I’m now considerably more confident that the vaccine worked (~90%). Also, that confidence is distributed over fewer possible worlds—e.g. based on the info in the latest version of the RadVac whitepaper, I now very much doubt that the vaccine will induce a response in the blood (unless it’s injected). I also now have very little weight on the possibility that it works for some people sometimes but didn’t work for me specifically, so additional dakka is not needed (at least for me).
The next section will be a bit more detail on the ELISA tests, for people who are curious about exactly how that sausage was made.
ELISA Tests
This section is an abbreviated chronology of what-I-saw and my reasoning about it; it’s intended to show how I came to the conclusions I did. I expect most people will not find it very interesting, but one of the benefits of blog posts is that I can show all the questionable decisions and opportunities for confirmation bias to sneak in, so that’s what I’m doing.
First, some background on how these tests work in theory. We start with a “high binding plate”—basically some plastic treated so that proteins/peptides stick to it.
That’s the plate; each of the holes is called a “well”, and is basically a mini-test-tube with a high-binding surface.
We add a solution of our peptides, and some of them stick to the surface. Next, we dump that solution out, leaving behind only the peptides which bound to the surface. We add some “binding solution”—in this case nonfat dry milk with a little detergent in it. The proteins in the binding solution fill whatever space on the surface was not taken up by the peptides.
Now, with the foundation in place, we build a tower. We add a nasal wash sample from my nose, which hopefully contains antibodies that bind to the peptides. Then we dump that out, leaving behind only the antibodies which bound to a peptide attached to the plate. Next we add the “secondary antibodies”, which bind to my antibodies and have an enzyme attached to them. Then we dump that solution out too. If all goes well, this leaves our “tower”: peptide bound to the plate, antibody bound to the peptide, secondary bound to the antibody.
The final step is to add some TMB solution. The enzyme attached to the secondary antibody will turn the TMB blue, so if we see blue after a few minutes, then we know the secondary antibodies are present, and hopefully that means the rest of the tower is present too.
So how does this play out in practice?
Here’s the first plate I ran:
Each row is a different sample (nasal wash and saliva from both myself and my girlfriend, blood from me, and one more concentrated nasal sample). Each column is a different peptide, with no-peptide control group in the right-most column. From this, we learn three things. First, there’s a lot of variation between samples. Second, the variation we’re seeing has nothing to do with the peptides. Third, and most important, the negative controls contain an awful lot of blue. So this does not match our theoretical picture of building-a-tower; somehow, something in the sample is sticking in the plate, and it’s sticking to something other than the peptide.
Next came a fair bit of trying stuff. I tried both more concentrated peptides, and dosing myself with RADVAC the day before taking a sample in order to induce more antibody production, both in hopes of increasing the signal enough to overcome the noise. Results were basically similar. I tried a bunch of different blocking agents, without any peptides at all, to see which gave the least-blue negative controls—dried milk was actually one of the best, although egg whites were better. So I tried using egg whites for the blocking solution, and results were basically similar. I ran another plate just comparing various negative controls, without any major new insight.
In general, I still saw some samples produce generally-more or generally-less blue, across both experimental and negative control groups, for no clear reason. Generally, things varied more from test-to-test and sample-to-sample than within similar treatment groups on the same sample/test—i.e. the noise is mostly systematic.
The main result from all this was that wells with no sample were pretty consistently not blue. Whatever causes my no-peptide negative controls to turn blue, it definitely involves something in the sample binding to something other than the peptide.
I did some reading online and some thinking about how to reduce the sample binding in the negative control group. I shifted to a mental model where a significant fraction of the peptide/binding agent on the plate was actually coming loose and being replaced by whatever was in solution. One way to reduce noise in the control, then, is to include a little binding agent in the antibody solution and secondary antibody solution—so if a space on the plate opens up, it will most likely be filled by the extra binding agent rather than the antibody. (This was included in the protocol, but listed as “optional”, and I didn’t understand before why it would be useful.) I also increased the amount of binding solution used (so that it covered the sides of the well more completely), and made extra sure to not accidentally use 100uL rather than 200 uL of binding solution (all of the other steps involve 100 uL in the well, so it’s an easy mistake to make if not paying attention—I think I probably made this mistake multiple times in earlier tests). Finally, I removed the wash step between peptide and binding solution—I see no way that that one could reduce noise, and I’d expect it to reduce the signal.
With all that in place, I ran two more tests. One saw generally very little blue, but there was more blue in the experimental wells than the controls:
Left-to-right, we have three no-peptide controls, then three experimental wells, then three more no-peptide controls, then three more experimental wells. There is definitely some blue in the no-peptide controls (especially the second well from the left), but there is generally more blue in the experimental wells. Note that this test was run with a sample which had been frozen; I think that’s the most likely cause of the generally-faint blueness.
Then we got the most promising result:
The group of five experimental wells in the upper right was visibly more blue than the corresponding no-peptide negative controls below them. (The group on the left is a sample which was frozen; so now we know to use fresh samples. This is another mistake I made multiple times in earlier tests.)
On the other hand, I’m still not able to get consistent results. I ran one more test the next day, just a single fresh sample with a whole bunch of experimental wells and no-peptide controls, and there was no visible difference between the controls and the experimental wells:
Experimental and control wells are in alternating rows. The last two columns are no-sample controls.
So, bottom line: the two positive results, especially the second, are more than I’d expect to see from noise after having run this many tests. But they’re still definitely not knock-down unambiguous evidence, and there’s still a lot of test-to-test variability which I’m unable to account for.
Going Forward
Mainstream vaccines will be available to the general populace here starting two weeks from today, so this project is probably reaching its end soon. We may run another test or two, but I probably won’t have another post. Conditional on any more tests, I will put up a shortform summarizing whatever results we see.
Another RadVac Testing Update
Previously: Making Vaccine, Commercial Antibody Test Results, Mini-Update
I’ve now run 9 ELISA tests. The main result is noise: negative controls are all over the map, sometimes very blue (i.e. positive), sometimes not blue at all. I did see more positive results in the experimental group than I’d expect from noise alone, but I haven’t gotten the noise to a point where results are consistently reproducible.
Meanwhile, I also ran one very simple test: I snorted a batch of the peptides, without the chitosan or anything else—just peptides in deionized water. Previously, on doses 3-6 of the vaccine, I had consistently been congested for a couple days after (and not congested the rest of the week), which strongly indicates an immune response. However, that response could have been to the chitosan or other contents of the vaccine, rather than the peptides. This test put that possibility to rest: after snorting just the peptides, I was very obviously congested for a couple days, in basically the same way as after the vaccine doses.
So thanks to that simple test, I personally am now pretty highly confident that I have an immune response to these peptides. Unfortunately it’s not as legible as an ELISA test, so you should not necessarily be quite as convinced by this.
Now, this still leaves the question of whether an immune response to these peptides translates into an immune response to COVID. It could be that e.g. the conformation of the peptides’ corresponding sequence within full COVID proteins is different enough that it doesn’t carry over. Personally, though, I consider this a much less likely failure mode, for two reasons. First, the white paper indicates that the peptides were chosen based on antibodies developed by people who actually had COVID. Second, whether antibodies against these peptides bind the real proteins is something which I would not expect to vary much from person to person, so if it’s worked for a few people it should work for everyone—and the whitepaper does indicate that multiple groups have seen positive results testing for binding against the full proteins.
None of this puts my confidence up close to 99%, but I’m now considerably more confident that the vaccine worked (~90%). Also, that confidence is distributed over fewer possible worlds—e.g. based on the info in the latest version of the RadVac whitepaper, I now very much doubt that the vaccine will induce a response in the blood (unless it’s injected). I also now have very little weight on the possibility that it works for some people sometimes but didn’t work for me specifically, so additional dakka is not needed (at least for me).
The next section will be a bit more detail on the ELISA tests, for people who are curious about exactly how that sausage was made.
ELISA Tests
This section is an abbreviated chronology of what-I-saw and my reasoning about it; it’s intended to show how I came to the conclusions I did. I expect most people will not find it very interesting, but one of the benefits of blog posts is that I can show all the questionable decisions and opportunities for confirmation bias to sneak in, so that’s what I’m doing.
First, some background on how these tests work in theory. We start with a “high binding plate”—basically some plastic treated so that proteins/peptides stick to it.
We add a solution of our peptides, and some of them stick to the surface. Next, we dump that solution out, leaving behind only the peptides which bound to the surface. We add some “binding solution”—in this case nonfat dry milk with a little detergent in it. The proteins in the binding solution fill whatever space on the surface was not taken up by the peptides.
Now, with the foundation in place, we build a tower. We add a nasal wash sample from my nose, which hopefully contains antibodies that bind to the peptides. Then we dump that out, leaving behind only the antibodies which bound to a peptide attached to the plate. Next we add the “secondary antibodies”, which bind to my antibodies and have an enzyme attached to them. Then we dump that solution out too. If all goes well, this leaves our “tower”: peptide bound to the plate, antibody bound to the peptide, secondary bound to the antibody.
The final step is to add some TMB solution. The enzyme attached to the secondary antibody will turn the TMB blue, so if we see blue after a few minutes, then we know the secondary antibodies are present, and hopefully that means the rest of the tower is present too.
So how does this play out in practice?
Here’s the first plate I ran:
Each row is a different sample (nasal wash and saliva from both myself and my girlfriend, blood from me, and one more concentrated nasal sample). Each column is a different peptide, with no-peptide control group in the right-most column. From this, we learn three things. First, there’s a lot of variation between samples. Second, the variation we’re seeing has nothing to do with the peptides. Third, and most important, the negative controls contain an awful lot of blue. So this does not match our theoretical picture of building-a-tower; somehow, something in the sample is sticking in the plate, and it’s sticking to something other than the peptide.
Next came a fair bit of trying stuff. I tried both more concentrated peptides, and dosing myself with RADVAC the day before taking a sample in order to induce more antibody production, both in hopes of increasing the signal enough to overcome the noise. Results were basically similar. I tried a bunch of different blocking agents, without any peptides at all, to see which gave the least-blue negative controls—dried milk was actually one of the best, although egg whites were better. So I tried using egg whites for the blocking solution, and results were basically similar. I ran another plate just comparing various negative controls, without any major new insight.
In general, I still saw some samples produce generally-more or generally-less blue, across both experimental and negative control groups, for no clear reason. Generally, things varied more from test-to-test and sample-to-sample than within similar treatment groups on the same sample/test—i.e. the noise is mostly systematic.
The main result from all this was that wells with no sample were pretty consistently not blue. Whatever causes my no-peptide negative controls to turn blue, it definitely involves something in the sample binding to something other than the peptide.
I did some reading online and some thinking about how to reduce the sample binding in the negative control group. I shifted to a mental model where a significant fraction of the peptide/binding agent on the plate was actually coming loose and being replaced by whatever was in solution. One way to reduce noise in the control, then, is to include a little binding agent in the antibody solution and secondary antibody solution—so if a space on the plate opens up, it will most likely be filled by the extra binding agent rather than the antibody. (This was included in the protocol, but listed as “optional”, and I didn’t understand before why it would be useful.) I also increased the amount of binding solution used (so that it covered the sides of the well more completely), and made extra sure to not accidentally use 100uL rather than 200 uL of binding solution (all of the other steps involve 100 uL in the well, so it’s an easy mistake to make if not paying attention—I think I probably made this mistake multiple times in earlier tests). Finally, I removed the wash step between peptide and binding solution—I see no way that that one could reduce noise, and I’d expect it to reduce the signal.
With all that in place, I ran two more tests. One saw generally very little blue, but there was more blue in the experimental wells than the controls:
Then we got the most promising result:
On the other hand, I’m still not able to get consistent results. I ran one more test the next day, just a single fresh sample with a whole bunch of experimental wells and no-peptide controls, and there was no visible difference between the controls and the experimental wells:
So, bottom line: the two positive results, especially the second, are more than I’d expect to see from noise after having run this many tests. But they’re still definitely not knock-down unambiguous evidence, and there’s still a lot of test-to-test variability which I’m unable to account for.
Going Forward
Mainstream vaccines will be available to the general populace here starting two weeks from today, so this project is probably reaching its end soon. We may run another test or two, but I probably won’t have another post. Conditional on any more tests, I will put up a shortform summarizing whatever results we see.