So they used only 400 IU, and even then the breast cancer incidence was 528 vs 546 for 400 IU vs 0 IU.
The difference in incidence was not statistically significant.
This is primitive frequentist statistics.
If we could repeat this same experiment with a very similar distribution of patients and other factors (or if the trial was still ongoing), and it came down to a bet on which group would have higher cancer deaths—would you accept an even money bet that the placebo group would have less cancer deaths? ie—would you bet against me?
Whether 3% more deaths in the placebo group is significant or not for a sample size of 1000, and an average supplement level which is only 10% of what the vit D experts believe is required to replace sunlight depends completely on your priors and your statistical model. There is nothing innate or inherently correct about the standard puny little models used to determine ‘statistical significance’.
Thinking in terms of prediction markets helps clarify these issues—the best model is the one that ends up sitting on the big heap pile of money.
Random chance could easily result in this level of variation.
Sure—or maybe not. It is still evidence that favors a vit D effect for breast cancer.
However, the authors are not unaware of this and point out that many women in the trial were supplementing with additional Vitamin D, and after analyzing that, there was still no detectable effect:
Half of the women reported taking an additional 400 IU, which—even with full compliance—only raises the average up to 600 IU.
The main key point is that the range of measured OHD levels was between 30 to 67 ish and didn’t have a strong correlation to the supplement—they failed to raise the OHD levels. That is key. All of the studies which show a health effect also significantly shift the OHD levels. If your supplement isn’t shifting OHD levels much, why would think it would do much of anything?
The general theory predicts health effects in 1.) certain genetic subpopulations, who 2.) have unnaturally low levels (say <30 for caucasians), and 3. then raise those levels up to evolutionary ideal for their genetics (perhaps 50-60 for caucasians). The exact numbers are just hyperparameters—the point of the studies should be to refine them into a predictive model.
But you cannot argue that the researchers haven’t done their homework or that they aren’t aware of what their data is capable or incapable of showing.
The completely honest conclusion for their abstract should have been something like this:
Conclusions Calcium and low doses of vitamin D supplementation did not significantly reduce (> 3%) invasive breast cancer incidence in postmenopausal women. In addition, 25-hydroxyvitamin D levels were not associated with subsequent breast cancer risk after we adjusted for body mass index. It is difficult to make definitive conclusions from these findings.
Compare that to the conclusion they actually wrote. Then the telephone game begins, and people begin quoting this study as one which somehow ‘proves’ vit D supplementation has no effect on breast cancer . …
I’m sorry, I’m really not interested in getting into an ideological debate or whether frequentist or bayesian statistics is “better”—if you think that frequentist methods are worthless, then the inferential gap is too wide to begin to bridge.
Agreed—I do believe that Frequentist methods are primitive compared to modern machine learning.
Also, I don’t even have a strong opinion on whether a few years of vit D supplementation in the elderly is going to make a big difference in many health outcomes like cancer risk—the correct comparison is between a lifetime of adequate vit D levels vs a lifetime of inadequacy. I don’t suspect that correcting it late in life is going to avoid most of the cancer risk.
This is primitive frequentist statistics.
If we could repeat this same experiment with a very similar distribution of patients and other factors (or if the trial was still ongoing), and it came down to a bet on which group would have higher cancer deaths—would you accept an even money bet that the placebo group would have less cancer deaths? ie—would you bet against me?
Whether 3% more deaths in the placebo group is significant or not for a sample size of 1000, and an average supplement level which is only 10% of what the vit D experts believe is required to replace sunlight depends completely on your priors and your statistical model. There is nothing innate or inherently correct about the standard puny little models used to determine ‘statistical significance’.
Thinking in terms of prediction markets helps clarify these issues—the best model is the one that ends up sitting on the big heap pile of money.
Sure—or maybe not. It is still evidence that favors a vit D effect for breast cancer.
Half of the women reported taking an additional 400 IU, which—even with full compliance—only raises the average up to 600 IU.
The main key point is that the range of measured OHD levels was between 30 to 67 ish and didn’t have a strong correlation to the supplement—they failed to raise the OHD levels. That is key. All of the studies which show a health effect also significantly shift the OHD levels. If your supplement isn’t shifting OHD levels much, why would think it would do much of anything?
The general theory predicts health effects in 1.) certain genetic subpopulations, who 2.) have unnaturally low levels (say <30 for caucasians), and 3. then raise those levels up to evolutionary ideal for their genetics (perhaps 50-60 for caucasians). The exact numbers are just hyperparameters—the point of the studies should be to refine them into a predictive model.
The completely honest conclusion for their abstract should have been something like this:
Conclusions Calcium and low doses of vitamin D supplementation did not significantly reduce (> 3%) invasive breast cancer incidence in postmenopausal women. In addition, 25-hydroxyvitamin D levels were not associated with subsequent breast cancer risk after we adjusted for body mass index. It is difficult to make definitive conclusions from these findings.
Compare that to the conclusion they actually wrote. Then the telephone game begins, and people begin quoting this study as one which somehow ‘proves’ vit D supplementation has no effect on breast cancer . …
If you want to bet there the VITAL study going on: http://www.vitalstudy.org/VitalSigns.html 25,000 subjects 2000 UI Vitamin D3 and upcoming mortality data.
I also created a prediction book entry a while ago: http://predictionbook.com/predictions/14426
I’m sorry, I’m really not interested in getting into an ideological debate or whether frequentist or bayesian statistics is “better”—if you think that frequentist methods are worthless, then the inferential gap is too wide to begin to bridge.
Agreed—I do believe that Frequentist methods are primitive compared to modern machine learning.
Also, I don’t even have a strong opinion on whether a few years of vit D supplementation in the elderly is going to make a big difference in many health outcomes like cancer risk—the correct comparison is between a lifetime of adequate vit D levels vs a lifetime of inadequacy. I don’t suspect that correcting it late in life is going to avoid most of the cancer risk.