I don’t see any obvious way in which this would be “cheating”.
Oh, that’s easy: publication bias. If the original studies report only the measures which reached a cutoff, and the null is always true, then since their measures will generally all be on the same subjects/with the same n, their effect sizes will have to be fairly similar* and I’d expect the i^2 to be low even as the results are meaningless.
* since p is just a function of sample size & effect size, and the p threshold is fixed by convention at 0.05, and sample size n is pretty much the same across all measures—since why would you recruit a subject and then not get as much data as possible and omit lots of subjects? - only measurements with effect sizes big enough to cross the p with the fixed n will be reported.
While if each particular measure was done separately as a bunch of univariate or multivariate meta-analyses, they’d have to get access to the original data or they’d be able to see the publication bias on a measure by measure basis.
Or it might be that each measure has a weighted effect size of zero, it’s just that each study is biased towards a different measure, and so its ‘overall’ estimate is positive even though if we had combined each measure with all its siblings, every single one would net to zero.
Maybe I’m wrong about these speculations. But I hope you see why I feel uncomfortable with this ‘lump everything remotely similar together’ approach and would like to see what meta-analytic experts say about the approach.
Oh, that’s easy: publication bias. If the original studies report only the measures which reached a cutoff, and the null is always true, then since their measures will generally all be on the same subjects/with the same n, their effect sizes will have to be fairly similar* and I’d expect the i^2 to be low even as the results are meaningless.
* since p is just a function of sample size & effect size, and the p threshold is fixed by convention at 0.05, and sample size n is pretty much the same across all measures—since why would you recruit a subject and then not get as much data as possible and omit lots of subjects? - only measurements with effect sizes big enough to cross the p with the fixed n will be reported.
While if each particular measure was done separately as a bunch of univariate or multivariate meta-analyses, they’d have to get access to the original data or they’d be able to see the publication bias on a measure by measure basis.
Or it might be that each measure has a weighted effect size of zero, it’s just that each study is biased towards a different measure, and so its ‘overall’ estimate is positive even though if we had combined each measure with all its siblings, every single one would net to zero.
Maybe I’m wrong about these speculations. But I hope you see why I feel uncomfortable with this ‘lump everything remotely similar together’ approach and would like to see what meta-analytic experts say about the approach.
That’s a great point, I hadn’t been thinking about that. It amplifies the publication bias by a lot.