The controversy about hormone replacement therapy is fascinating as a case study. Until 2002, essentially all women who reached menopause got medical advise to start taking pills containing horse estrogen. It was very widely believed that this would reduce their risk of having a heart attack. This belief primarily based on biological plausibility: Estrogen is known to reduce cholesterol, and cholesterol is believed to increase the risk of heart disease. Also, there were many observational studies that seemingly suggested that women who took hormone replacement therapy (HRT) had less risk of heart disease. (In my view, this was not surprising: Observational studies always show what the investigators expect to find.)
In 2002, the Women’s Health Initiative randomized trial was stopped early because it showed that estrogen replacement therapy actually substantially increased the risk of having a heart attack. Overnight, the medical establishment stopped recommending estrogen for menopausal women. But a perhaps more important consequence was that many clinicians stopped trusting observational studies altogether. In my opinion, this was mostly a good thing.
The largest observational study to show a protective effect of estrogen the Nurses Health Study. In 2008, my thesis advisor Miguel Hernan re-analyzed this dataset using Jamie Robins’ g-methods (which are equivalent to Pearl), and was essentially able to reproduce the results of the WHI trial. Miguel’s paper uses valid methods and gets the correct results. In my view, this shows that the new methods might work, but the paper would have meant much more if it was published prior to the randomized trials.
Miguel and Jamie’s paper sparked off a very interesting methodological debate with the original investigators at the Nurses Health Study, who are still clinging to their original analysis. See http://www.ncbi.nlm.nih.gov/pubmed/18813017 .
Many people still believe that Estrogen/HRT is beneficial. The most popular theory is that WHI recruited too many old women (sometimes in their 90s!) and that estrogen is harmful if given that long after menopause. A new randomized trial which is limited to women at menopause is currently being conducted. A second theory is that the results in the trial were due to differences in statin usage. I analyzed the second theory for my doctoral thesis, but found that this had negligible impact on the results.
It is also interesting to note that while it is true that the trial found that estrogen increased the risk of heart disease, it also showed a (non-significant) reduction in all-cause mortality. So the increased risk of cardiovascular disease didn’t even result in more deaths. Presumably, people care more about all-cause mortality than heart attacks. However, since it was “non-significant”, not even the most dedicated proponents of estrogen treatment ever point out this fact.
A side question, prompted by an amusing factoid in the Hernan paper: ”...we restricted the population to women who had reported plausible energy intakes (2510 –14,640 kJ/d)”.
In the statistical analysis in this paper, and also as a general practice in medical publications based on questionnaire data, are there adjustments for uncertainty in the questionnaire responses?
When you have a data point that says, for example, that person #12345 reports her caloric intake as 4,000 calories/day, do you take it as a hard precise number, or do you take it as an imprecise estimate with its own error which propagates into the model uncertainty, etc.?
Keyword is “measurement error.” People think hard about this. Anders_H knows this paper in a lot more detail than I do, but I expect these particular authors to be careful.
This issue is also related to “missing data.” What you see might be different from the underlying truth in systematic ways, e.g. you get systematic bias in your data, and you need to deal with that. This is also related to that causal inference stuff I keep going on about.
Keyword is “measurement error.” People think hard about this.
People like engineers and physicists think a lot about this. I am not sure that medical researchers think a lot about this. The usual (easy) way is to throw out unreasonable-looking responses during the data cleaning and then take what remains as rock-solid. Accepting that your independent variables are uncertain leads to a lot of inconvenient problems (starting with the OLS regression not being a theoretically-correct form any more).
What you see might be different from the underlying truth in systematic ways, e.g. you get systematic bias in your data, and you need to deal with that.
Yes, that’s another can of worms. In some areas (e.g. self-reported food intake) the problem is so blatant and overwhelming that you have to deal with it, but if it looks minor not many people want to bother.
The paper you are referring to is “Observational Studies Analyzed Like Randomized Experiments: An application to Postmenopausal Hormone Therapy and Coronary Heart Disease” by Hernan et al. It is available at https://cdn1.sph.harvard.edu/wp-content/uploads/sites/343/2013/03/observational-studies.pdf
The controversy about hormone replacement therapy is fascinating as a case study. Until 2002, essentially all women who reached menopause got medical advise to start taking pills containing horse estrogen. It was very widely believed that this would reduce their risk of having a heart attack. This belief primarily based on biological plausibility: Estrogen is known to reduce cholesterol, and cholesterol is believed to increase the risk of heart disease. Also, there were many observational studies that seemingly suggested that women who took hormone replacement therapy (HRT) had less risk of heart disease. (In my view, this was not surprising: Observational studies always show what the investigators expect to find.)
In 2002, the Women’s Health Initiative randomized trial was stopped early because it showed that estrogen replacement therapy actually substantially increased the risk of having a heart attack. Overnight, the medical establishment stopped recommending estrogen for menopausal women. But a perhaps more important consequence was that many clinicians stopped trusting observational studies altogether. In my opinion, this was mostly a good thing.
The largest observational study to show a protective effect of estrogen the Nurses Health Study. In 2008, my thesis advisor Miguel Hernan re-analyzed this dataset using Jamie Robins’ g-methods (which are equivalent to Pearl), and was essentially able to reproduce the results of the WHI trial. Miguel’s paper uses valid methods and gets the correct results. In my view, this shows that the new methods might work, but the paper would have meant much more if it was published prior to the randomized trials.
Miguel and Jamie’s paper sparked off a very interesting methodological debate with the original investigators at the Nurses Health Study, who are still clinging to their original analysis. See http://www.ncbi.nlm.nih.gov/pubmed/18813017 .
Many people still believe that Estrogen/HRT is beneficial. The most popular theory is that WHI recruited too many old women (sometimes in their 90s!) and that estrogen is harmful if given that long after menopause. A new randomized trial which is limited to women at menopause is currently being conducted. A second theory is that the results in the trial were due to differences in statin usage. I analyzed the second theory for my doctoral thesis, but found that this had negligible impact on the results.
It is also interesting to note that while it is true that the trial found that estrogen increased the risk of heart disease, it also showed a (non-significant) reduction in all-cause mortality. So the increased risk of cardiovascular disease didn’t even result in more deaths. Presumably, people care more about all-cause mortality than heart attacks. However, since it was “non-significant”, not even the most dedicated proponents of estrogen treatment ever point out this fact.
A side question, prompted by an amusing factoid in the Hernan paper: ”...we restricted the population to women who had reported plausible energy intakes (2510 –14,640 kJ/d)”.
In the statistical analysis in this paper, and also as a general practice in medical publications based on questionnaire data, are there adjustments for uncertainty in the questionnaire responses?
When you have a data point that says, for example, that person #12345 reports her caloric intake as 4,000 calories/day, do you take it as a hard precise number, or do you take it as an imprecise estimate with its own error which propagates into the model uncertainty, etc.?
Keyword is “measurement error.” People think hard about this. Anders_H knows this paper in a lot more detail than I do, but I expect these particular authors to be careful.
This issue is also related to “missing data.” What you see might be different from the underlying truth in systematic ways, e.g. you get systematic bias in your data, and you need to deal with that. This is also related to that causal inference stuff I keep going on about.
People like engineers and physicists think a lot about this. I am not sure that medical researchers think a lot about this. The usual (easy) way is to throw out unreasonable-looking responses during the data cleaning and then take what remains as rock-solid. Accepting that your independent variables are uncertain leads to a lot of inconvenient problems (starting with the OLS regression not being a theoretically-correct form any more).
Yes, that’s another can of worms. In some areas (e.g. self-reported food intake) the problem is so blatant and overwhelming that you have to deal with it, but if it looks minor not many people want to bother.
Clinicians do not, “methodology people” (who often partner up with “domain experts”) to do data analysis, absolutely do.
Yes, I was told the full gory details of this story (not going to repeat it here). Thanks for sharing this!
By the way, are you at Stanford now? I should find a way to drop by, Jacob’s there too.