RE Robin Hanson’s post on the health insurance study, ‘Most of the p-values weren’t statistically significant, therefore there’s no effect’ is not reasoning that I generally find very convincing.
I’d rather have them combine all the health outcome measures into a single estimate of overall health. Then we can look at the point estimate of the difference in health between getting insurance vs. not, test if it’s statistically significant, look at whether the effect size is too tiny to care much about or meaningful, and look at whether all of the effect sizes in the 95% confidence interval are too tiny to care about or if some of them are meaningful. Maybe even do a cost-benefit estimate, trying to get the units into something more like DALYs per dollar.
(IIRC, the Oregon study that Hanson mentions in his post did something kinda like this, and the point estimate was that people who got insurance had better health, this was not statistically significant, but the point estimate was large enough to care about.)
Looking at the new paper, it does do a few different analyses of effects on health outcomes, but doesn’t say much about them. They call health outcomes a secondary measure and give just these two paragraphs of results in the text of the paper:
Access to insurance had few significant effects on health in either survey (Table 5). Having measured (a) 3 parameters (direct/indirect/total) for (b) 3 ITT and one TOT effect for (c) 82 specified outcomes over 2 surveys, only 3 (0.46% of all estimated coefficients concerning health outcomes) were significant after multiple-testing adjustments. (As Table A8 shows, 55 parameters (8.38%) are significant if we do not adjust for multiple-testing.) We cannot reject the hypothesis that the distribution of p-values from these estimates is consistent with no differences (P=0.31). We also find no effect of access on our summary index of health outcomes (Table A6 and Table A7).
Care should be taken in interpreting the insignificant health effects observed. Perhaps the effect of hospital care on measured outcomes is too small to translate into health improvements that we have power to detect despite our substantial sample size (Das, Hammer et al. 2008). Moreover, confidence intervals reported in Table A6 and Table A7 suggest that medically significant effects for many outcomes cannot be ruled out. On average, the absolute value of an estimated ITT (TOT) effect for an outcome equals 11% (8.8%) the standard deviation of the outcome. Finally, given the low premiums for RSBY insurance, it would require a rather precise nearly zero estimate of health effects to rule out that government spending on freely provided insurance was not cost-effective.
So they did calculate a single overall effect on health, which they say elsewhere was “the average of z-scores for individual health outcomes” (that seems like an adequate way to do it but not the best, since it ignores the importance & noisiness of each measure, and the correlations between measures). They say that the effect on that overall health index was not statistically significant, but they don’t say anything about the effect size or confidence interval (maybe there’s something in the appendix but I can’t find find anything about the index in Table A6 or A7).
RE Robin Hanson’s post on the health insurance study, ‘Most of the p-values weren’t statistically significant, therefore there’s no effect’ is not reasoning that I generally find very convincing.
I’d rather have them combine all the health outcome measures into a single estimate of overall health. Then we can look at the point estimate of the difference in health between getting insurance vs. not, test if it’s statistically significant, look at whether the effect size is too tiny to care much about or meaningful, and look at whether all of the effect sizes in the 95% confidence interval are too tiny to care about or if some of them are meaningful. Maybe even do a cost-benefit estimate, trying to get the units into something more like DALYs per dollar.
(IIRC, the Oregon study that Hanson mentions in his post did something kinda like this, and the point estimate was that people who got insurance had better health, this was not statistically significant, but the point estimate was large enough to care about.)
Looking at the new paper, it does do a few different analyses of effects on health outcomes, but doesn’t say much about them. They call health outcomes a secondary measure and give just these two paragraphs of results in the text of the paper:
So they did calculate a single overall effect on health, which they say elsewhere was “the average of z-scores for individual health outcomes” (that seems like an adequate way to do it but not the best, since it ignores the importance & noisiness of each measure, and the correlations between measures). They say that the effect on that overall health index was not statistically significant, but they don’t say anything about the effect size or confidence interval (maybe there’s something in the appendix but I can’t find find anything about the index in Table A6 or A7).