Thanks for bringing up the topic of vaccine passports and doing it in a thoughtful way. Before getting started on the stuff I can actually claim to have some insight into, just a few quick points of order about the Excelsior Pass:
It’s not really a vaccine passport, or even an immunity passport, it’s a broad “good to go” credential based on whether you’ve been vaccinated or you’ve had a recent negative test.
As best I can tell the 30 day limit is just an artifact of when you download it. You can reset the clock by downloading it again the next day, and there’s no limit on how many times you download it.
Also an important point about the extent to which there might be something rolled out at the federal level: the Biden administration has already indicated that except possibly for international travel, they don’t intend to mandate anything of the sort for domestic use. Exact language matters a lot here. The feds are using the term “digital vaccine certificate”, rather than “vaccine passport”, and I think the connotation is important. The former suggests something more agnostic as far as use-case. The latter suggests something much more rigid, and something that’s only really appropriate for travel (as opposed to something you’d have to carry around with you anywhere you go). According to Jeffrey Zients, “any solutions in this area should be simple, free, open source, accessible to people both digitally and on paper, and designed from the start to protect people’s privacy.” I boldfaced “open source” because I really think that’s key here. People, and by extension governments composed of people, make mistakes. An idea can be good-ish, and people can put in work developing an app, but as long as we keep this open source we can salvage any work that’s been done without throwing the baby out with the bathwater. Not surprisingly, this is a position shared by the American Civil Liberties Union.
Now to the aspect that I can actually claim insight into—“fraud concerns”. They’re valid. Before delving into why they’re valid, I want to pose the question: what’s the “end goal” of vaccine passports? Is it to
a) Ensure with 100% accuracy who is or isn’t vaccinated? or
b) Maximize the percentage of vaccinated people at a given public venue?
What I want to argue for is that (a) is a fool’s errand and that there are better, more cost-effective ways of getting (b). In the process I’m also going to show why “open source” is a key characteristic of any digital vaccine certificate.
About me: I’ve been a civil servant working in state government for almost a decade. In that time I’ve been exposed to a lot of different things that state government does to enforce a lot of different things that could be viewed as rough analogs to what’s being suggested with vaccine passports. The recurring theme is that fraud and forgery are common, and become more so when people have a high incentive to commit fraud and forgery for personal financial gain (or avoiding personal financial losses). Indeed, in a managerial accounting class one might learn about the “fraud triangle”:
fraud triangle
It’s not hard to see that all three of the above elements exist for “vaccine passports”, were we to focus on that as our main tool for accomplishing the basic goal of (b) above. Here are some of the examples where fraud is surprisingly common (warning: a lot of the links below are to New York Times articles, and you may vary well blow through your monthly quota if you don’t have a subscription—high quality information isn’t cheap).
1. Fish About 40% of all fish is mislabeled. The problem periodically resurfaces when it’s examined by investigative journalists or law enforcement, and there’s very little evidence that it’s going away, despite our best efforts to enforce it. One of the more disturbing forms of this is when escolar is passed off as tuna. Escolar is considered somewhat of a delicacy, but when consumed in the same quantities as tuna, it causes explosive diarrhea.
2. Taxi meters—unscrupulous taxi drivers can and do tamper with their meters. A noteworthy instance of widespread fraud occurred in 2010. Cyrus Vance uncovered it in a massive sting operation that presumably involved a lot of overtime.
3. Gasoline: This doesn’t happen as much as the above forms of fraud, but only because we have a government agency devoted to its prevention. But suffice to say that without a large number of government agents acting as secret shoppers, this kind of thing would happen:
4. Supermarket scales: another thing that state governments need to keep an eye on. Because people cheat.
5. Firewood: Moving firewood is a pretty big no-no everywhere in the country, and for good reason—it helps move pests from point A to point B, potentially leading to devastating blights on local flora. All levels of government are involved in enforcing this, both with secret shoppers and with periodic checkpoints.
6. Fraudulent motor vehicle inspections: motor vehicle inspection shops have a fairly high motivation to pass paying customers on their inspections. Occasionally this is unearthed by undercover operations.
There are loads more examples—the key takeaway is that people can and do cheat. Maybe not a lot of people, but enough people do it enough that enforcing it is (pardon my language) hard as fuck. It’s a game of whack-a-mole with a limited number of resources, and it involves sting operations and secret shoppers. And it’s difficult to see how simply putting something on a smartphone changes the fundamental dynamic of one human checking another human’s credentials. Even when merchants do catch attempts at forgery and act in good faith, things can go sideways, as is abundantly clear from the unfolding, increasingly agonizing story about George Floyd. The fact that the fake was easy to spot wasn’t the issue; the escalation was the problem. Anyone tasked with enforcing this needs to be carefully trained on de-escalation tactics, and in a country as awash in guns as ours, that’s a pretty big ask.
So that’s where I’m coming from with my skepticism about the feasibility about any large-scale deployment of “vaccine passports”. It seems to me that the game isn’t worth the candle, for this particular method of obtaining the goal of (b).
So before I get into two alternatives that I thought of—I think we can all agree that we should be hoping that we’re able to convince everyone to get vaccinated quickly and efficiently enough for their own sake that the need for all of this is obviated. It’s good to see that the Biden administration is making a concerted effort to do exactly that.
It should be noted that both of the alternatives I’m suggesting could easily incorporate some sort of digital vaccine certificate, just not used in the way that some people have suggested for vaccine passports.
Batch verification of vaccination status: Here’s how this could work. Say someone (the “organizer”) wants to facilitate a group of people (“attendees”) in one place. Instead of verifying the vaccination status of each individual at the door, the organizer turns to a third party to check the vaccination status of the attendees, but with a catch. The third party only reports back the number of vaccinated attendees. There’s no reason that a “digital vaccine certificate” couldn’t be used to make this easier for the third party (which is why I emphasized the importance of this software being open source). What kind of third party? Either a local health department or a licensed ticket reseller might do the trick. In the latter case, the event organizer could sell the tickets to the licensed ticket reseller, who would check the percentage of vaccinated prospective attendees. If the numbers don’t work, the event is canceled and the reseller absorbs the loss, which is easier for ticket resellers, since that’s sort of what they’re for anyway. The important point is that there’s very little motivation either for fraud or violent escalation with this approach. Moreover, it acts as a better safeguard for personal information. Finally, I suspect this would also be better from a contact tracing perspective. A contact tracer would only have to deal with the ticket reseller. To reiterate, digital vaccine certificates could very much have a role to play in a system like this; “vaccine passports” not so much.
Incentives: By now everyone has heard about the free doughnuts at Krispy Kreme, and I’d love to know if it supports the math that I’m about to outline below. As far as I can tell, reputable sources confirm that incentives work. Even the studies that claim they don’t work still end up demonstrating that they increase the number of people who get vaccinated, even when they don’t meet their targets, which suggests that if the financial incentives were larger, even more people would get vaccinated. And they don’t all have to be monetary. A baseball stadium could offer discounted season passes to people who get vaccinated—heck, they could send them directly to whoever’s doing the vaccinating (quick, CVS/Walgreen’s/NYU Langone/whoever is reading this—get a ticket resale license!). What happens next? Well, vaccinated people will be more likely to step out. And here’s where Bayes’ Theorem comes into play. Specifically consider the following probabilities:
P(A)= probability of someone attending an event P(V)= probability that someone happens to be vaccinated P(A∩V)= probability that someone is attending an event and is vaccinated P(A|V)= probability that someone is at an event given that they’re vaccinated P(V|A)= probability that someone is vaccinated given that they’re at an event
Sorry, I seem to have reached my character limit, so I’m continuing this in a reply to myself—I think this is important.
There’s no reason to think that incentives wouldn’t serve to maximize P(A|V). Bayes’ Theorem relates P(V|A) and P(A|V) as follows:
P(V|A)=P(A|V)⋅P(V)P(A)
The beauty of an incentive system is it would maximize both factors in the numerator while the denominator can stay more or less fixed (e.g. by the capacity of the venue). Same reduction in risk for people attending, at minimal cost.
Again, I want to emphasize that I can imagine a role for digital vaccine certificates in such a system, but the idea of vaccine passports presented at the door, just seems untenable.
It turns out that non-profits in New York State don’t need to obtain licenses to resell tickets. So it would be pretty easy for an event organizer in New York City to, say, sell a bunch of tickets in bulk to a large hospital system that uses MyChart. The hospital system can then resell the tickets to its own patients, as a promotion—possibly with a corporate sponsorship.
I would encourage you to make this a top-level post, I think there’s a lot of very useful content here and I’d like to be able to comment / refer back to it. I’m especially interested in exploring why these particular areas have so much fraud relative to other areas slash whether this is true—one question is whether these are areas where we call people who lie or misrepresent out as committing fraud, whereas in other places maybe we don’t as much do so.
The solutions on the other hand don’t seem viable to me. E.g. having a system where it will tell you how many out of X or more people are vaccinated, but won’t tell you if 1 particular person is vaccinated, sounds like something you do in math team practice or when nerd sniping at a party to figure out how to figure out exactly who is vaccinated, and/or a way to start a lot of fights and have a lot of really bad free rider problems and game theory experiments that mostly don’t end so well. Fascinating stuff, though. I’m curious how you think this functions in practice if there’s a bar on directly checking individuals, under your proposals.
Incentives are great and would certainly help with the ‘fuzzy math’ of having groups contain more vaccinated people, slash getting more people vaccinated, but I don’t think there’s any political/social ability to notice that going from 30% to 70% vaccinated in groups is ‘good enough’ in some sense and we should be OK with it, I think it needs to be effectively 100% or things won’t actually happen. And yes, you can say ‘but fraud!’ but in some sense that serves the function of letting everyone pretend it’s 100% slash not feel responsible for the fact that it’s not 100% or for the people still vulnerable.
So I guess another approach that would make vaccine passports palatable to everyone would be if we just went ahead and gave everybody something similar to the Excelsior pass (either on their phone or printed), where we’d attempt to implement that underlying logic of “x% vaccinated or y% capacity” in real time. The venue would stop allowing new people in when neither of those conditions are met, and nobody would see anybody’s personal information. And it would take away any motive for the prospective attendee to cheat. This could also be leveraged for contact tracing, and perhaps the expectation would be that you get a test if you don’t feel well, or if a contact tracer tracks you down.
I don’t think it would be too difficult to deploy something like this (all-in-one contact tracing/vaccination tracking/compliance app)
Yeah, so I guess my point is that in the spirit of “less wrong”, making a beeline for aggregate statistics appears to me to be the “least wrong”.
There’s also somewhat promising evidence that there’s going to be enough self selection that Bayes’ Theorem will have our backs even without incentives. Kinda like how there are a lot of uh, people like me in movie theaters on December 25th.
This is fascinating. I expect many readers of Less Wrong would be interested in top-level posts about what the world looks like from the perspective of a state government civil servant.
Thanks. That means a lot to me. I feel like a lot of things depend on “who’s in the room” when decisions are made, and all too often it’s the people who are stuck with implementing things that are left out.
Thanks for bringing up the topic of vaccine passports and doing it in a thoughtful way. Before getting started on the stuff I can actually claim to have some insight into, just a few quick points of order about the Excelsior Pass:
It’s not really a vaccine passport, or even an immunity passport, it’s a broad “good to go” credential based on whether you’ve been vaccinated or you’ve had a recent negative test.
As best I can tell the 30 day limit is just an artifact of when you download it. You can reset the clock by downloading it again the next day, and there’s no limit on how many times you download it.
Also an important point about the extent to which there might be something rolled out at the federal level: the Biden administration has already indicated that except possibly for international travel, they don’t intend to mandate anything of the sort for domestic use. Exact language matters a lot here. The feds are using the term “digital vaccine certificate”, rather than “vaccine passport”, and I think the connotation is important. The former suggests something more agnostic as far as use-case. The latter suggests something much more rigid, and something that’s only really appropriate for travel (as opposed to something you’d have to carry around with you anywhere you go). According to Jeffrey Zients, “any solutions in this area should be simple, free, open source, accessible to people both digitally and on paper, and designed from the start to protect people’s privacy.” I boldfaced “open source” because I really think that’s key here. People, and by extension governments composed of people, make mistakes. An idea can be good-ish, and people can put in work developing an app, but as long as we keep this open source we can salvage any work that’s been done without throwing the baby out with the bathwater. Not surprisingly, this is a position shared by the American Civil Liberties Union.
Now to the aspect that I can actually claim insight into—“fraud concerns”. They’re valid. Before delving into why they’re valid, I want to pose the question: what’s the “end goal” of vaccine passports? Is it to
a) Ensure with 100% accuracy who is or isn’t vaccinated? or
b) Maximize the percentage of vaccinated people at a given public venue?
What I want to argue for is that (a) is a fool’s errand and that there are better, more cost-effective ways of getting (b). In the process I’m also going to show why “open source” is a key characteristic of any digital vaccine certificate.
About me: I’ve been a civil servant working in state government for almost a decade. In that time I’ve been exposed to a lot of different things that state government does to enforce a lot of different things that could be viewed as rough analogs to what’s being suggested with vaccine passports. The recurring theme is that fraud and forgery are common, and become more so when people have a high incentive to commit fraud and forgery for personal financial gain (or avoiding personal financial losses). Indeed, in a managerial accounting class one might learn about the “fraud triangle”:
It’s not hard to see that all three of the above elements exist for “vaccine passports”, were we to focus on that as our main tool for accomplishing the basic goal of (b) above. Here are some of the examples where fraud is surprisingly common (warning: a lot of the links below are to New York Times articles, and you may vary well blow through your monthly quota if you don’t have a subscription—high quality information isn’t cheap).
1. Fish About 40% of all fish is mislabeled. The problem periodically resurfaces when it’s examined by investigative journalists or law enforcement, and there’s very little evidence that it’s going away, despite our best efforts to enforce it. One of the more disturbing forms of this is when escolar is passed off as tuna. Escolar is considered somewhat of a delicacy, but when consumed in the same quantities as tuna, it causes explosive diarrhea.
2. Taxi meters—unscrupulous taxi drivers can and do tamper with their meters. A noteworthy instance of widespread fraud occurred in 2010. Cyrus Vance uncovered it in a massive sting operation that presumably involved a lot of overtime.
3. Gasoline: This doesn’t happen as much as the above forms of fraud, but only because we have a government agency devoted to its prevention. But suffice to say that without a large number of government agents acting as secret shoppers, this kind of thing would happen:
4. Supermarket scales: another thing that state governments need to keep an eye on. Because people cheat.
5. Firewood: Moving firewood is a pretty big no-no everywhere in the country, and for good reason—it helps move pests from point A to point B, potentially leading to devastating blights on local flora. All levels of government are involved in enforcing this, both with secret shoppers and with periodic checkpoints.
6. Fraudulent motor vehicle inspections: motor vehicle inspection shops have a fairly high motivation to pass paying customers on their inspections. Occasionally this is unearthed by undercover operations.
7. Underage sale of alcohol and tobacco: One of the exceptions to child labor laws is the use of minors to pose as underage customers trying to buy tobacco and alcohol.
There are loads more examples—the key takeaway is that people can and do cheat. Maybe not a lot of people, but enough people do it enough that enforcing it is (pardon my language) hard as fuck. It’s a game of whack-a-mole with a limited number of resources, and it involves sting operations and secret shoppers. And it’s difficult to see how simply putting something on a smartphone changes the fundamental dynamic of one human checking another human’s credentials. Even when merchants do catch attempts at forgery and act in good faith, things can go sideways, as is abundantly clear from the unfolding, increasingly agonizing story about George Floyd. The fact that the fake was easy to spot wasn’t the issue; the escalation was the problem. Anyone tasked with enforcing this needs to be carefully trained on de-escalation tactics, and in a country as awash in guns as ours, that’s a pretty big ask.
So that’s where I’m coming from with my skepticism about the feasibility about any large-scale deployment of “vaccine passports”. It seems to me that the game isn’t worth the candle, for this particular method of obtaining the goal of (b).
So before I get into two alternatives that I thought of—I think we can all agree that we should be hoping that we’re able to convince everyone to get vaccinated quickly and efficiently enough for their own sake that the need for all of this is obviated. It’s good to see that the Biden administration is making a concerted effort to do exactly that.
It should be noted that both of the alternatives I’m suggesting could easily incorporate some sort of digital vaccine certificate, just not used in the way that some people have suggested for vaccine passports.
Batch verification of vaccination status: Here’s how this could work. Say someone (the “organizer”) wants to facilitate a group of people (“attendees”) in one place. Instead of verifying the vaccination status of each individual at the door, the organizer turns to a third party to check the vaccination status of the attendees, but with a catch. The third party only reports back the number of vaccinated attendees. There’s no reason that a “digital vaccine certificate” couldn’t be used to make this easier for the third party (which is why I emphasized the importance of this software being open source). What kind of third party? Either a local health department or a licensed ticket reseller might do the trick. In the latter case, the event organizer could sell the tickets to the licensed ticket reseller, who would check the percentage of vaccinated prospective attendees. If the numbers don’t work, the event is canceled and the reseller absorbs the loss, which is easier for ticket resellers, since that’s sort of what they’re for anyway. The important point is that there’s very little motivation either for fraud or violent escalation with this approach. Moreover, it acts as a better safeguard for personal information. Finally, I suspect this would also be better from a contact tracing perspective. A contact tracer would only have to deal with the ticket reseller. To reiterate, digital vaccine certificates could very much have a role to play in a system like this; “vaccine passports” not so much.
Incentives: By now everyone has heard about the free doughnuts at Krispy Kreme, and I’d love to know if it supports the math that I’m about to outline below. As far as I can tell, reputable sources confirm that incentives work. Even the studies that claim they don’t work still end up demonstrating that they increase the number of people who get vaccinated, even when they don’t meet their targets, which suggests that if the financial incentives were larger, even more people would get vaccinated. And they don’t all have to be monetary. A baseball stadium could offer discounted season passes to people who get vaccinated—heck, they could send them directly to whoever’s doing the vaccinating (quick, CVS/Walgreen’s/NYU Langone/whoever is reading this—get a ticket resale license!). What happens next? Well, vaccinated people will be more likely to step out. And here’s where Bayes’ Theorem comes into play. Specifically consider the following probabilities:
P(A)= probability of someone attending an event
P(V)= probability that someone happens to be vaccinated
P(A∩V)= probability that someone is attending an event and is vaccinated
P(A|V)= probability that someone is at an event given that they’re vaccinated
P(V|A)= probability that someone is vaccinated given that they’re at an event
Sorry, I seem to have reached my character limit, so I’m continuing this in a reply to myself—I think this is important.
There’s no reason to think that incentives wouldn’t serve to maximize P(A|V). Bayes’ Theorem relates P(V|A) and P(A|V) as follows:
P(V|A)=P(A|V)⋅P(V)P(A)
The beauty of an incentive system is it would maximize both factors in the numerator while the denominator can stay more or less fixed (e.g. by the capacity of the venue). Same reduction in risk for people attending, at minimal cost.
Again, I want to emphasize that I can imagine a role for digital vaccine certificates in such a system, but the idea of vaccine passports presented at the door, just seems untenable.
It turns out that non-profits in New York State don’t need to obtain licenses to resell tickets. So it would be pretty easy for an event organizer in New York City to, say, sell a bunch of tickets in bulk to a large hospital system that uses MyChart. The hospital system can then resell the tickets to its own patients, as a promotion—possibly with a corporate sponsorship.
I would encourage you to make this a top-level post, I think there’s a lot of very useful content here and I’d like to be able to comment / refer back to it. I’m especially interested in exploring why these particular areas have so much fraud relative to other areas slash whether this is true—one question is whether these are areas where we call people who lie or misrepresent out as committing fraud, whereas in other places maybe we don’t as much do so.
The solutions on the other hand don’t seem viable to me. E.g. having a system where it will tell you how many out of X or more people are vaccinated, but won’t tell you if 1 particular person is vaccinated, sounds like something you do in math team practice or when nerd sniping at a party to figure out how to figure out exactly who is vaccinated, and/or a way to start a lot of fights and have a lot of really bad free rider problems and game theory experiments that mostly don’t end so well. Fascinating stuff, though. I’m curious how you think this functions in practice if there’s a bar on directly checking individuals, under your proposals.
Incentives are great and would certainly help with the ‘fuzzy math’ of having groups contain more vaccinated people, slash getting more people vaccinated, but I don’t think there’s any political/social ability to notice that going from 30% to 70% vaccinated in groups is ‘good enough’ in some sense and we should be OK with it, I think it needs to be effectively 100% or things won’t actually happen. And yes, you can say ‘but fraud!’ but in some sense that serves the function of letting everyone pretend it’s 100% slash not feel responsible for the fact that it’s not 100% or for the people still vulnerable.
So I guess another approach that would make vaccine passports palatable to everyone would be if we just went ahead and gave everybody something similar to the Excelsior pass (either on their phone or printed), where we’d attempt to implement that underlying logic of “x% vaccinated or y% capacity” in real time. The venue would stop allowing new people in when neither of those conditions are met, and nobody would see anybody’s personal information. And it would take away any motive for the prospective attendee to cheat. This could also be leveraged for contact tracing, and perhaps the expectation would be that you get a test if you don’t feel well, or if a contact tracer tracks you down.
I don’t think it would be too difficult to deploy something like this (all-in-one contact tracing/vaccination tracking/compliance app)
Yeah, so I guess my point is that in the spirit of “less wrong”, making a beeline for aggregate statistics appears to me to be the “least wrong”.
There’s also somewhat promising evidence that there’s going to be enough self selection that Bayes’ Theorem will have our backs even without incentives. Kinda like how there are a lot of uh, people like me in movie theaters on December 25th.
This is fascinating. I expect many readers of Less Wrong would be interested in top-level posts about what the world looks like from the perspective of a state government civil servant.
Thanks. That means a lot to me. I feel like a lot of things depend on “who’s in the room” when decisions are made, and all too often it’s the people who are stuck with implementing things that are left out.