Some problems with evidence-based aid
In Against Health: How Health Became the New Morality (Biopolitics, Medicine, Technoscience, and Health in the 21st Century) , there’s an essay “Against Global Health?” by Vincanne Adams.
The introduction was a bit of a slog, but I think the point was that if “health” is defined as something which can and should be given to people regardless of what they think in the matter, the results can be presumptuous.
While she may be overly invested in the way things are usually done, she brings up some disquieting points about the ways insisting on double-blind tests can go wrong. For example, she mentions a project to evaluate training in safe infant delivery techniques in Tibet which was scuttled because not enough women were dying to get a good power calculation.
She describes experiments which are done in isolated communities because that’s where the researchers can be reasonably sure that the subjects don’t have alternative sources of care which would foul up the double-blinding. This does seem like a drunk and lamp post problem. [1] More generally, the concern is that actual care is delayed in the search for perfect information. Admittedly, it’s hard to be sure about how to balance searching for information with taking action, but it strikes me as a problem that’s worth some thought rather than just assuming that double-blinding is a reliable improvement.
[1] Is there a standard LW term for searching where it’s easy to search rather than where the answer is likely to be?
I’m really confused by this. Both examples sound like exactly what you should be doing.
If the Tibetan study was to determine how many people died in childbirth, and they refused to publish because not enough people were dying to support their preferred narrative, that would be bad. But the way you wrote it sounds like it was to determine differences in deaths across two conditions, but there weren’t enough deaths to tell. In that case, stopping the study instead of publishing results that are probably false is the best possible course of action.
Likewise, doing studies in isolated villages is a good way to control for confounding variables. If you wanted to measure effectiveness of a cancer drug, you do it on people who are only taking the one drug, not people taking lots of drugs that might differ across groups. If you’re measuring life expectancy in AIDS, you do it on patients who only have AIDS, not patients who have AIDS and also cancer. If a physicist is measuring the radioactivity of U-235, ey makes sure to use a purified sample of U-235, not a sample mixed with U-238. I don’t even see what this has to do with double-blinding (which is about placebos and making sure subjects don’t know what group they’re in). This is just a basic attempt to isolate the experimental variable. I agree that given infinite time and resources they should try follow-up studies in less isolated villages to see if the results carry over, but in reality this seems like the best way to do things.
Am I misunderstanding the book’s points?
I suspect that the Tibet study issue could be solved with better statistical methodology (i.e. using a Bayesian framework and including semi-realistic prior information). However, setting that aside, I agree completely.
I don’t know of a standard LW term, but why not “lamp posting”?
“Googling”?
Which could be amplified as ngramming—the comments have a lot about how sloppy the results are.
This suggests that the old methods were safe enough and this project was not worth it. Statistical significance is not the right measure, but this was probably handled correctly.
Yes, isolated communities are easier to study, a kind of lamppost. If you think that all the studies will work in the isolated communities, that is exactly the lamppost error and there’s no point to the studies. I would call that the search for perfect certainty of the answers to wrong questions (or limited questions), not perfect information, which I would expect to describe expensive studies of other communities. But I don’t think all the studies will be positive. I expect many interventions to fail even in the most controlled settings; they probably won’t work anywhere else, either. This is quite valuable information, even if it gives a pass to interventions that won’t work elsewhere.
Possibly—but some of the training methods might lead to fewer complications.
I don’t know of a standard LW term, but I suggest “lamp posting.”
The old story is that a drunk man is on his hands and knees near a lamp post. A passerby asks him what he’s doing, and he replies that he is looking for his watch, “which I lost in that alley over there.” The passerby inquires, “Then why aren’t you looking in the alley?” “It’s too dark there; here is where the light is.”