No, it’s more subtle than that. I think it’s more clearly stated in terms of effect sizes. (Down with null hypothesis significance testing!) The study measured the average effect of food dye on hyperactivity in the population and showed it was not distinguishable from zero. The quoted conclusion makes the unfounded assumption that that all children can be characterized by that small average effect. This ignores unmeasured confounders, which another way of phrasing PhilGoetz’s correct (CORRECT, PEOPLE, CORRECT!) point.
The document I linked mentions doing a “sensitivity analysis for the possibility of unmeasured confounding, to see the sorts of changes one could expect if there were such a confounder.” In the above study (assuming PhilGoetz described it correctly; I haven’t read the original paper), the data permitted such a sensitivity analysis. It would have given an upper bound for the effect of the unmeasured confounder as a function of an assumed prevalence of the confound in the population. (A smaller assumed prevalence gives a larger upper bound.) But if you don’t even notice that it’s possible for children to have heterogeneous responses to the treatment, you’ll never even think of doing such a sensitivity analysis.
No, it’s more subtle than that. I think it’s more clearly stated in terms of effect sizes. (Down with null hypothesis significance testing!) The study measured the average effect of food dye on hyperactivity in the population and showed it was not distinguishable from zero. The quoted conclusion makes the unfounded assumption that that all children can be characterized by that small average effect. This ignores unmeasured confounders, which another way of phrasing PhilGoetz’s correct (CORRECT, PEOPLE, CORRECT!) point.
The document I linked mentions doing a “sensitivity analysis for the possibility of unmeasured confounding, to see the sorts of changes one could expect if there were such a confounder.” In the above study (assuming PhilGoetz described it correctly; I haven’t read the original paper), the data permitted such a sensitivity analysis. It would have given an upper bound for the effect of the unmeasured confounder as a function of an assumed prevalence of the confound in the population. (A smaller assumed prevalence gives a larger upper bound.) But if you don’t even notice that it’s possible for children to have heterogeneous responses to the treatment, you’ll never even think of doing such a sensitivity analysis.