I’m pointing out what seem to me to be large and important holes in your argument.
To an objection of the form “You have given no good reason to think Y follows from X”, it is not reasonable to respond with “You need to give a specific example of how you can have X and not Y, with realistic numbers in it”.
I claim that you have given no reason to think that if there’s a lot of good to be done at $5k per life-equivalent then there is necessarily an experiment that it’s feasible for (say) GiveWell to conduct that would do something like eliminating all malaria deaths in Madagascar for a year. You’ve just said that obviously there must be.
I reject any norms that say that in that situation anyone saying that your reasoning has gaps in is obliged to show concrete counterexamples.
However, because I’m an obliging sort of chap, let’s have a go at constructing one and see what happens. (But, for the avoidance of doubt, I am not conceding that if my specific counterexample turns out not to work then it means your claim is right and mine is wrong. Of course it’s possible that you know ahead of time that I can’t construct a working counterexample, on account of having a better understanding than mine of the situation—but, again, in that case communicating that better understanding should be part of your argument.) I’ll look at Madagascar since that’s the country you mentioned specifically.
[EDITED to add:] Although the foregoing paragraph talks about “constructing a counterexample”, in fact what I did in the following paragraphs is just to make some guesses about numbers and see where they lead; I wasn’t trying to pick numbers that are maximally persuasive or anything.
So, first of all let’s find some numbers. Madagascar has a population of about 26 million. Malaria is the 7th most common cause of death there. If I’m reading the stats correctly, about 10% of the population has malaria and they have about 6k deaths per year. Essentially the entire population is considered at risk. At present Madagascar gets about $50M/year of malaria-fighting from the rest of the world. Insecticide-treated bed nets allegedly reduce the risk of getting malaria by ~70% compared with not having them; it’s not clear to me how that’s defined, but let’s suppose it’s per year. The statistics I’ve seen differ somewhat in their estimates of what fraction of the Madagascan population has access to bed nets; e.g., in this document from the WHO plot E on page 85 seems to show only ~5% of the population with access to either bed nets or indoor spraying; the table on page 117 says 6%; but then another table on page 122 estimates ~80% of households have at least one net and ~44% have at least one per two people. I guess maybe most Madagascan households have a great many people? These figures are much lower in Madagascar than in most of Africa; I don’t know why. It seems reasonable to guess that bed net charities expect it to be more expensive, more difficult or less effective in Madagascar than in the other places where they have distributed more nets, but again even if this is correct I don’t know what the underlying reasons are. I observe that several African countries have a lot more malaria deaths per unit population; e.g., Niger has slightly fewer people than Madagascar but nearly 3x as many malaria deaths. (And also about 3x as many people with malaria.) So maybe bed net distribution focuses on those countries?
So, my first observation is that this is all consistent with the possbility that the number of lives saveable in Madagascar at ~$5k/life is zero, because of some combination of { lower prevalence of malaria, higher cost of distributing nets, lower effectiveness of nets } there compared with, say, Niger or the DRC. This seems like the simplest explanation of the fact that Madagascar has surprisingly few bed nets per person, and it seems consistent with the fact that, while it certainly has a severe malaria problem, it has substantially less malaria per person than many other African countries. Let’s make a handwavy guess that the effectiveness per dollar of bednets in Madagascar is half what it is in the countries with the best effectiveness-per-dollar opportunities, which conditional on that $5k/life-equivalent figure would mean $10k/life-equivalent.
Now, as to fatality: evidently the huge majority of people with malaria do not die in any given year. (~2.5M cases, ~6k deaths.) Malaria is a serious disease even when it doesn’t kill you. Back of envelope: suppose deaths from malaria in Madagascar cost 40 QALYs each (life expectancy in Madagascar is ~66y, many malaria deaths are of young children but not all, there’s a lot of other disease in Madagascar and I guess quality of life is often poor, handwave handwave; 40 QALYs seems like the right ballpark) and suppose having malaria but not dying costs 0.05 QALYs per year (it puts you completely out of action some of the time, makes you feel ill a lot more of the time, causes mental distress, sometimes does lasting organ damage, etc.; again I’m making handwavy estimates). Then every year Madagascar loses ~125k QALYs to nonfatal malaria and ~240k QALYs to fatal malaria. Those numbers are super-inexact and all I’m really comfortable concluding here is that the two are comparable. I guess (though I don’t know) that bednets are somewhere around equally effective in keeping adults and children from getting malaria, and that there isn’t any correlation between preventability-by-bednet and severity in any particular case; so I expect the benefits of bednets in death-reduction and other-illness-reduction to, again, be comparable. I believe death, when it occurs, is commonly soon after infection, but the other effects commonly persist for a long time. I’m going to guess that 3⁄4 of the effects of a change in bednet use happen within ~ a year, with a long tail for the rest.
So, let’s put that together a bit. Most of the population is not currently protected by bednets. If they suddenly were then we might expect a ~70% reduction in new malaria cases that year, for those protected by the nets. Best case, that might mean a ~70% reduction in malaria deaths that year; presumably the actual figure is a bit less because some malaria deaths happen longer after infection. Call it 60%. Reduction in malaria harm that year would be more like 50%. Cost would be $10k per life-equivalent saved. Total cost somewhere on the order of $50M, a substantial fraction of e.g. AMF’s total assets.
Another way to estimate the cost: GiveWell estimates that AMF’s bednet distribution costs somewhere around $4.50 per net. So one net per person in Madagascar is $100M or so.
But that’s only ~60% of the deaths; you wanted a nice clear-cut experiment that got rid of all the malaria deaths in Madagascar for one year. And indeed cutting deaths by 60% would not necessarily be conclusive, because the annual variation in malaria cases in Madagascar seems to be large and so is the uncertainty in counting those cases. In the 2010-2017 period the point estimates in the document I linked above have been as low as ~2200 and as high as ~7300; the error bars each year go from just barely above zero to nearly twice the point estimate. (These uncertainties are much larger, incidentally, than in many other African countries with similar malaria rates, which seems consistent with there being something about Madagascar that makes treatment and/or measurement harder than other African countries.)
To get rid of all (or nearly all) the deaths in one year, presumably you need to eliminate infection that happens while people aren’t sleeping under their bed nets, and to deal with whatever minority of people are unwilling or unable to use bed nets. Those seem like harder problems. I think countries that have eliminated malaria have done it by eliminating the mosquitoes that spread it, which is a great long-term solution if you can do it but much harder than distributing bed nets. So my best guess is that if you want to get rid of all the malaria, even for one year, you will have to spend an awful lot more per life-equivalent saved that year; I would be unsurprised by 10x as much, not that surprised by 100x, and not altogether astonished if it turned out that no one actually knows how to do it for any amount of money. It might still be worth it if the costs are large—the future effects are large if you can eliminate malaria from a place permanently. (Which might be easier in Madagascar than in many other African countries, since it’s an island.) But it puts the costs out of the range of “things existing EA charities could easily do to prove a point”. And it’s a Gates Foundation sort of project, not an AMF one, and indeed as I understand it the Gates Foundation is putting a lot of money into investigating ways to eliminate malaria.
Tentative conclusion: It’s not a all obvious to me that this sort of experiment would be worth while. For “only” an amount of money comparable to the total assets of the Against Malaria Foundation, it looks like it might be possible to somewhat-more-than-halve malaria deaths in Madagascar for one year (and reduce ongoing malaria a bit in subsequent years). The expected benefits of doing this would be substantially less than those of distributing bed nets in the probably-more-cost-effective other places where organizations like AMF are currently putting them. Given how variable the prevalence of malaria is in Madagascar, and how uncertain the available estimates of that prevalence seem to be, it is not clear that doing this would be anything like conclusive evidence that bednet distribution is as effective as it’s claimed to be. (All of the foregoing is conditional on the assumption that it is as effective as claimed.) To get such conclusive evidence, it would be necessary to do things radically different from, and probably far more expensive than, bednet distribution; organizations like AMF would have neither the expertise nor the resources to do that.
I am not very confident about any of the numbers above (other than “easy” ones like the population of Madagascar), and all my calculations are handwavy estimates (because there’s little point doing anything more careful when the underlying numbers are so doubtful). But what those calculations suggest to me is that, whether or not doing the sort of experiment you propose would be a good idea, it doesn’t seem to be an obviously good idea (since, in particular, my current best estimate is that it would not be a good idea). Therefore, unless I am shown compelling evidence pointing in a different direction, I cannot take seriously the claim that EA organizations that aren’t doing such experiments show thereby that they don’t believe that there is large scope for doing good at a price on the order of $5k per life-equivalent.
You’ve given a lot of details specifically about Madagascar, but not actually responded to the substantive argument in the post. What global picture does this correspond to, under which the $5k per life saved figure is still true and meaningful? I don’t see how the existence of somewhere for which no lives can be saved for $5k makes that claim any more plausible.
Your claim, as I understood it—which maybe I didn’t, because you have been frustratingly vague about your own argument at the same time as demanding ever-increasing amounts of detail from anyone who questions it—was that if the $5k-per-life-equivalent figure were real then there “should” be some experiment that could be done “in a well-defined area like Madagascar” that would be convincing enough to be a good use of the (large) resources it would cost.
I suggest that the scenario I described above is obviously consistent with a $5k-per-life-equivalent figure in the places where bednets are most effective per unit spent. I assume you picked Madagascar because (being isolated, fairly small, etc.) it would be a good place for an experiment.
If you think it is not credible that any global picture makes the $5k figure “true and meaningful” then it is up to you to give a good argument for that. So far, it seems to me that you have not done so; you have asserted that if it were true then EA organizations should be running large-scale experiments to prove it, but you haven’t offered any credible calculations or anything to show that if the $5k figure were right then doing such experiments would be a good use of the available resources, and my back-of-envelope calculations above suggest that in the specific place you proposed, namely Madagascar, they quite likely wouldn’t be.
Perhaps I’m wrong. I often am. But I think you need to provide more than handwaving here. Show us your detailed models and calculations that demonstrate that if the $5k figure is anywhere near right then EA organizations should be acting very differently from how they actually are acting. Stop making grand claims and then demanding that other people do the hard work of giving quantitative evidence that you’re wrong, when you yourself haven’t done the hard work of giving quantitative evidence that you’re right.
Once again I say: what you are doing here is not what arguing in good faith usually looks like.
As you point out, you’re making entirely nonspecific claims. This is a waste of everyone’s time; please stop doing so here.
I’m pointing out what seem to me to be large and important holes in your argument.
To an objection of the form “You have given no good reason to think Y follows from X”, it is not reasonable to respond with “You need to give a specific example of how you can have X and not Y, with realistic numbers in it”.
I claim that you have given no reason to think that if there’s a lot of good to be done at $5k per life-equivalent then there is necessarily an experiment that it’s feasible for (say) GiveWell to conduct that would do something like eliminating all malaria deaths in Madagascar for a year. You’ve just said that obviously there must be.
I reject any norms that say that in that situation anyone saying that your reasoning has gaps in is obliged to show concrete counterexamples.
However, because I’m an obliging sort of chap, let’s have a go at constructing one and see what happens. (But, for the avoidance of doubt, I am not conceding that if my specific counterexample turns out not to work then it means your claim is right and mine is wrong. Of course it’s possible that you know ahead of time that I can’t construct a working counterexample, on account of having a better understanding than mine of the situation—but, again, in that case communicating that better understanding should be part of your argument.) I’ll look at Madagascar since that’s the country you mentioned specifically.
[EDITED to add:] Although the foregoing paragraph talks about “constructing a counterexample”, in fact what I did in the following paragraphs is just to make some guesses about numbers and see where they lead; I wasn’t trying to pick numbers that are maximally persuasive or anything.
So, first of all let’s find some numbers. Madagascar has a population of about 26 million. Malaria is the 7th most common cause of death there. If I’m reading the stats correctly, about 10% of the population has malaria and they have about 6k deaths per year. Essentially the entire population is considered at risk. At present Madagascar gets about $50M/year of malaria-fighting from the rest of the world. Insecticide-treated bed nets allegedly reduce the risk of getting malaria by ~70% compared with not having them; it’s not clear to me how that’s defined, but let’s suppose it’s per year. The statistics I’ve seen differ somewhat in their estimates of what fraction of the Madagascan population has access to bed nets; e.g., in this document from the WHO plot E on page 85 seems to show only ~5% of the population with access to either bed nets or indoor spraying; the table on page 117 says 6%; but then another table on page 122 estimates ~80% of households have at least one net and ~44% have at least one per two people. I guess maybe most Madagascan households have a great many people? These figures are much lower in Madagascar than in most of Africa; I don’t know why. It seems reasonable to guess that bed net charities expect it to be more expensive, more difficult or less effective in Madagascar than in the other places where they have distributed more nets, but again even if this is correct I don’t know what the underlying reasons are. I observe that several African countries have a lot more malaria deaths per unit population; e.g., Niger has slightly fewer people than Madagascar but nearly 3x as many malaria deaths. (And also about 3x as many people with malaria.) So maybe bed net distribution focuses on those countries?
So, my first observation is that this is all consistent with the possbility that the number of lives saveable in Madagascar at ~$5k/life is zero, because of some combination of { lower prevalence of malaria, higher cost of distributing nets, lower effectiveness of nets } there compared with, say, Niger or the DRC. This seems like the simplest explanation of the fact that Madagascar has surprisingly few bed nets per person, and it seems consistent with the fact that, while it certainly has a severe malaria problem, it has substantially less malaria per person than many other African countries. Let’s make a handwavy guess that the effectiveness per dollar of bednets in Madagascar is half what it is in the countries with the best effectiveness-per-dollar opportunities, which conditional on that $5k/life-equivalent figure would mean $10k/life-equivalent.
Now, as to fatality: evidently the huge majority of people with malaria do not die in any given year. (~2.5M cases, ~6k deaths.) Malaria is a serious disease even when it doesn’t kill you. Back of envelope: suppose deaths from malaria in Madagascar cost 40 QALYs each (life expectancy in Madagascar is ~66y, many malaria deaths are of young children but not all, there’s a lot of other disease in Madagascar and I guess quality of life is often poor, handwave handwave; 40 QALYs seems like the right ballpark) and suppose having malaria but not dying costs 0.05 QALYs per year (it puts you completely out of action some of the time, makes you feel ill a lot more of the time, causes mental distress, sometimes does lasting organ damage, etc.; again I’m making handwavy estimates). Then every year Madagascar loses ~125k QALYs to nonfatal malaria and ~240k QALYs to fatal malaria. Those numbers are super-inexact and all I’m really comfortable concluding here is that the two are comparable. I guess (though I don’t know) that bednets are somewhere around equally effective in keeping adults and children from getting malaria, and that there isn’t any correlation between preventability-by-bednet and severity in any particular case; so I expect the benefits of bednets in death-reduction and other-illness-reduction to, again, be comparable. I believe death, when it occurs, is commonly soon after infection, but the other effects commonly persist for a long time. I’m going to guess that 3⁄4 of the effects of a change in bednet use happen within ~ a year, with a long tail for the rest.
So, let’s put that together a bit. Most of the population is not currently protected by bednets. If they suddenly were then we might expect a ~70% reduction in new malaria cases that year, for those protected by the nets. Best case, that might mean a ~70% reduction in malaria deaths that year; presumably the actual figure is a bit less because some malaria deaths happen longer after infection. Call it 60%. Reduction in malaria harm that year would be more like 50%. Cost would be $10k per life-equivalent saved. Total cost somewhere on the order of $50M, a substantial fraction of e.g. AMF’s total assets.
Another way to estimate the cost: GiveWell estimates that AMF’s bednet distribution costs somewhere around $4.50 per net. So one net per person in Madagascar is $100M or so.
But that’s only ~60% of the deaths; you wanted a nice clear-cut experiment that got rid of all the malaria deaths in Madagascar for one year. And indeed cutting deaths by 60% would not necessarily be conclusive, because the annual variation in malaria cases in Madagascar seems to be large and so is the uncertainty in counting those cases. In the 2010-2017 period the point estimates in the document I linked above have been as low as ~2200 and as high as ~7300; the error bars each year go from just barely above zero to nearly twice the point estimate. (These uncertainties are much larger, incidentally, than in many other African countries with similar malaria rates, which seems consistent with there being something about Madagascar that makes treatment and/or measurement harder than other African countries.)
To get rid of all (or nearly all) the deaths in one year, presumably you need to eliminate infection that happens while people aren’t sleeping under their bed nets, and to deal with whatever minority of people are unwilling or unable to use bed nets. Those seem like harder problems. I think countries that have eliminated malaria have done it by eliminating the mosquitoes that spread it, which is a great long-term solution if you can do it but much harder than distributing bed nets. So my best guess is that if you want to get rid of all the malaria, even for one year, you will have to spend an awful lot more per life-equivalent saved that year; I would be unsurprised by 10x as much, not that surprised by 100x, and not altogether astonished if it turned out that no one actually knows how to do it for any amount of money. It might still be worth it if the costs are large—the future effects are large if you can eliminate malaria from a place permanently. (Which might be easier in Madagascar than in many other African countries, since it’s an island.) But it puts the costs out of the range of “things existing EA charities could easily do to prove a point”. And it’s a Gates Foundation sort of project, not an AMF one, and indeed as I understand it the Gates Foundation is putting a lot of money into investigating ways to eliminate malaria.
Tentative conclusion: It’s not a all obvious to me that this sort of experiment would be worth while. For “only” an amount of money comparable to the total assets of the Against Malaria Foundation, it looks like it might be possible to somewhat-more-than-halve malaria deaths in Madagascar for one year (and reduce ongoing malaria a bit in subsequent years). The expected benefits of doing this would be substantially less than those of distributing bed nets in the probably-more-cost-effective other places where organizations like AMF are currently putting them. Given how variable the prevalence of malaria is in Madagascar, and how uncertain the available estimates of that prevalence seem to be, it is not clear that doing this would be anything like conclusive evidence that bednet distribution is as effective as it’s claimed to be. (All of the foregoing is conditional on the assumption that it is as effective as claimed.) To get such conclusive evidence, it would be necessary to do things radically different from, and probably far more expensive than, bednet distribution; organizations like AMF would have neither the expertise nor the resources to do that.
I am not very confident about any of the numbers above (other than “easy” ones like the population of Madagascar), and all my calculations are handwavy estimates (because there’s little point doing anything more careful when the underlying numbers are so doubtful). But what those calculations suggest to me is that, whether or not doing the sort of experiment you propose would be a good idea, it doesn’t seem to be an obviously good idea (since, in particular, my current best estimate is that it would not be a good idea). Therefore, unless I am shown compelling evidence pointing in a different direction, I cannot take seriously the claim that EA organizations that aren’t doing such experiments show thereby that they don’t believe that there is large scope for doing good at a price on the order of $5k per life-equivalent.
You’ve given a lot of details specifically about Madagascar, but not actually responded to the substantive argument in the post. What global picture does this correspond to, under which the $5k per life saved figure is still true and meaningful? I don’t see how the existence of somewhere for which no lives can be saved for $5k makes that claim any more plausible.
Your claim, as I understood it—which maybe I didn’t, because you have been frustratingly vague about your own argument at the same time as demanding ever-increasing amounts of detail from anyone who questions it—was that if the $5k-per-life-equivalent figure were real then there “should” be some experiment that could be done “in a well-defined area like Madagascar” that would be convincing enough to be a good use of the (large) resources it would cost.
I suggest that the scenario I described above is obviously consistent with a $5k-per-life-equivalent figure in the places where bednets are most effective per unit spent. I assume you picked Madagascar because (being isolated, fairly small, etc.) it would be a good place for an experiment.
If you think it is not credible that any global picture makes the $5k figure “true and meaningful” then it is up to you to give a good argument for that. So far, it seems to me that you have not done so; you have asserted that if it were true then EA organizations should be running large-scale experiments to prove it, but you haven’t offered any credible calculations or anything to show that if the $5k figure were right then doing such experiments would be a good use of the available resources, and my back-of-envelope calculations above suggest that in the specific place you proposed, namely Madagascar, they quite likely wouldn’t be.
Perhaps I’m wrong. I often am. But I think you need to provide more than handwaving here. Show us your detailed models and calculations that demonstrate that if the $5k figure is anywhere near right then EA organizations should be acting very differently from how they actually are acting. Stop making grand claims and then demanding that other people do the hard work of giving quantitative evidence that you’re wrong, when you yourself haven’t done the hard work of giving quantitative evidence that you’re right.
Once again I say: what you are doing here is not what arguing in good faith usually looks like.