They’re self reporting, which is to say they have an incentive to represent their results as better than they really are. Until someone else performs that research, this is the highest-quality source of data available. If GiveWell thinks it checks out, I’m strongly inclined to believe it.
Changes in mortality are really difficult to disentangle, even over very long timespans (which DMI doesn’t have yet). Accepting their decision to study changes in self-reported behaviors, they’ve structured their research in the best way possible (RCT).
Also, doesn’t their approach have an intuitive appeal? Figure out what sickens/kills people in a place, find the cheapest behavior-change-based intervention, and tell everyone to do it using conventional advertising channels. I’d be surprised if it didn’t have some measurable effect (which I won’t say about the vast majority of charities).
Telling people via mass media that they should behave better doesn’t have a great track record :-/
I think at least the prevailing view in public health is that it does. This report is a decade old, but was the clearest summary I could find on a quick search. They do emphasise that media campaigns are more likely to be effective when awareness is a major issue and when the desired behaviour change is not that large (both seem true in the case of DMI).
I don’t think the report you link to supports your claim. In particular, while it shows some evidence that mass-media campaigns can raise the awareness of an issue, there is no conclusive evidence that they make anyone actually change their behaviour. And that’s before we look at the cost-effectiveness of the whole thing.
They do give a few examples of changing behaviour. For instance:
However, a controlled trial of a TV advertising campaign in central and northern England provides evidence that mass media campaigns may be able to change behaviour. The campaign was effective in reducing smoking prevalence by about 1.2% over 18 months.
However I agree that it doesn’t have fantastic evidence of that. But most of my impressions of this come from talking to people who work in public health; my understanding is that at least in rich countries, properly targeted public health campaigns are actually very cost-effective. How this carries over to poor countries is another question, but as a baseline I’d at least assume it’s plausible.
They do give a few examples of changing behaviour.
One—which you quoted—and which they offset by the immediately following paragraph which says (emphasis mine):
A study found that the proportion of people who were knowledgeable about the new recommendations increased significantly after the campaign, although it was unclear whether it was TV advertising or other elements of the campaign that made the difference (Hillsdon et al., 2001). However, there was no evidence that the campaign raised levels of physical activity.
There are more examples on the following page (although they are all time series rather than controlled trials, the effect sizes are large enough that it is implausible that they all represent natural background shifts).
I certainly don’t think that all public health campaigns are effective, or that awareness always translates into action. I just thought that your statements sounded surprisingly negative about the possibility of them being cost-effective.
They’re self reporting, which is to say they have an incentive to represent their results as better than they really are. Until someone else performs that research, this is the highest-quality source of data available. If GiveWell thinks it checks out, I’m strongly inclined to believe it.
Changes in mortality are really difficult to disentangle, even over very long timespans (which DMI doesn’t have yet). Accepting their decision to study changes in self-reported behaviors, they’ve structured their research in the best way possible (RCT).
Also, doesn’t their approach have an intuitive appeal? Figure out what sickens/kills people in a place, find the cheapest behavior-change-based intervention, and tell everyone to do it using conventional advertising channels. I’d be surprised if it didn’t have some measurable effect (which I won’t say about the vast majority of charities).
That’s only part of it—the other part is Goodhart’s Law and the fact that they are trying really hard to produce good metrics.
That’s not a good reason to search for your keys under a streetlight.
No, not to me. Telling people via mass media that they should behave better doesn’t have a great track record :-/
Do you believe samples of self-reported behavior can’t be an informative proxy for harder population metrics, like morbidity or mortality?
They could be after you establish the relationship through empirical data.
I think at least the prevailing view in public health is that it does. This report is a decade old, but was the clearest summary I could find on a quick search. They do emphasise that media campaigns are more likely to be effective when awareness is a major issue and when the desired behaviour change is not that large (both seem true in the case of DMI).
I don’t think the report you link to supports your claim. In particular, while it shows some evidence that mass-media campaigns can raise the awareness of an issue, there is no conclusive evidence that they make anyone actually change their behaviour. And that’s before we look at the cost-effectiveness of the whole thing.
They do give a few examples of changing behaviour. For instance:
However I agree that it doesn’t have fantastic evidence of that. But most of my impressions of this come from talking to people who work in public health; my understanding is that at least in rich countries, properly targeted public health campaigns are actually very cost-effective. How this carries over to poor countries is another question, but as a baseline I’d at least assume it’s plausible.
One—which you quoted—and which they offset by the immediately following paragraph which says (emphasis mine):
Negative evidence is evidence, too.
There are more examples on the following page (although they are all time series rather than controlled trials, the effect sizes are large enough that it is implausible that they all represent natural background shifts).
I certainly don’t think that all public health campaigns are effective, or that awareness always translates into action. I just thought that your statements sounded surprisingly negative about the possibility of them being cost-effective.