(As is Simpson’s 2nd paradox, which is arguably worse.)
Correlations between studies, either from common confounding factors or from re-use of data.
External confounding variables.
The combination means that in the presence of a lot of data, there are often many competing hypotheses, many of which are rational under different assumptions. Some examples:
Two studies. One shows X with p=0.04, then the attempted replication with a larger group gets p=0.13.
Person A says that the original study was flawed and X is false.
Person B says that the replication was flawed and X is true.
Person C says that the replication wasn’t a replication, and X is likely true only for the original group, not the larger group.
Person D says that the replication didn’t have sufficient statistical power, and X is true.
Person E says that the original study hit Simpson’s paradox, and X is false (but may be true within subgroups).
Four hundred studies. Three hundred of which have p>0.05, the last hundred of which have p<0.05
Person A finds 19 of the p<0.05 studies in a row and stops looking, as they physically cannot read every study in existence and so must stop somewhere. Somewhere down the line they may happen across a p>0.05 study and reject it, as it’s obviously an outlier.
(This is unlikely if you were selecting truly random studies; good luck truly selecting random studies.)
Person B finds a meta-meta-study showing 5 of 20 meta-studies show X.
Person C finds a different meta-meta-study (that rejects 10 of the meta-studies due to arguably-overlapping datasets) showing 0 of 10 meta-studies show X.
Person D finds 2 studies showing p>0.05 and 1 showing p<0.05, looks into the studies, and notices that both p>0.05 studies are sponsored by Entity E that they distrust.
Person E finds 4 studies, digs into them all, and realizes that they are testing slightly different things and one of them has a typo in the abstract, and gives up.
Person F finds a news article about someone being forced to resign due to not retracting a study on X, calls the entire thing political and unknowable due to selection bias.
*****
(Fair warning: I use statistics a fair bit; I am not a statistician.)
*****
I notice all those ratios look the same and also that one of them is cancer.
I would be very cautious about confounding variables here. We’re now 2 years down the line from a bunch of routine medical checkups being cancelled or postponed.
Here’s one potential correlation path off the top of my head:
Assume: many people had medical checkups cancelled or postponed due to COVID.
Assume: routine medical checkups are done because they do, on average, help catch issues before they become a more major problem. (I have seen people challenge this; it’s certainly at least a plausible assumption.)
Assume: people with pre-existing medical conditions are more likely to have routine medical checkups. (Again, I can see people challenge this; it’s certainly at least a plausible assumption.)
Assume: people with pre-existing medical conditions are more likely to be infected with COVID (at least in the ‘was noticed enough to be included in the study’ sense.)
Result: {COVID infection} correlates with {pre-existing medical conditions} correlates with {having medical checkups} correlates with {medical checkups cancelled or postponed} correlates with {issues not dealt with before they become a more major problem}
This can be controlled for to an extent—although given the ubiquity of cancellations in some places I’m not sure how large a sample remains after you do the control. Did they control for this? I don’t know. And them not controlling for this would be consistent with their conclusions, at least at a quick glance.
(There are likely other correlation paths.)
(Anecdote: I know multiple people with cancer right now that “should have” been caught earlier, but was missed until recently due to postponed or cancelled medical appointments.)
A few other components that are worth noting (you tangentially touched on the topic, but didn’t cover these directly):
p<0.05 still means that 1 in 20 studies come to a different conclusion, even under optimistic assumptions.
Statistical power is a thing.
Berkson’s paradox is also a thing.
Simpson’s Paradox is also a thing.
(As is Simpson’s 2nd paradox, which is arguably worse.)
Correlations between studies, either from common confounding factors or from re-use of data.
External confounding variables.
The combination means that in the presence of a lot of data, there are often many competing hypotheses, many of which are rational under different assumptions. Some examples:
Two studies. One shows X with p=0.04, then the attempted replication with a larger group gets p=0.13.
Person A says that the original study was flawed and X is false.
Person B says that the replication was flawed and X is true.
Person C says that the replication wasn’t a replication, and X is likely true only for the original group, not the larger group.
Person D says that the replication didn’t have sufficient statistical power, and X is true.
Person E says that the original study hit Simpson’s paradox, and X is false (but may be true within subgroups).
Four hundred studies. Three hundred of which have p>0.05, the last hundred of which have p<0.05
Person A finds 19 of the p<0.05 studies in a row and stops looking, as they physically cannot read every study in existence and so must stop somewhere. Somewhere down the line they may happen across a p>0.05 study and reject it, as it’s obviously an outlier.
(This is unlikely if you were selecting truly random studies; good luck truly selecting random studies.)
Person B finds a meta-meta-study showing 5 of 20 meta-studies show X.
Person C finds a different meta-meta-study (that rejects 10 of the meta-studies due to arguably-overlapping datasets) showing 0 of 10 meta-studies show X.
Person D finds 2 studies showing p>0.05 and 1 showing p<0.05, looks into the studies, and notices that both p>0.05 studies are sponsored by Entity E that they distrust.
Person E finds 4 studies, digs into them all, and realizes that they are testing slightly different things and one of them has a typo in the abstract, and gives up.
Person F finds a news article about someone being forced to resign due to not retracting a study on X, calls the entire thing political and unknowable due to selection bias.
*****
(Fair warning: I use statistics a fair bit; I am not a statistician.)
*****
I would be very cautious about confounding variables here. We’re now 2 years down the line from a bunch of routine medical checkups being cancelled or postponed.
Here’s one potential correlation path off the top of my head:
Assume: many people had medical checkups cancelled or postponed due to COVID.
Assume: routine medical checkups are done because they do, on average, help catch issues before they become a more major problem. (I have seen people challenge this; it’s certainly at least a plausible assumption.)
Assume: people with pre-existing medical conditions are more likely to have routine medical checkups. (Again, I can see people challenge this; it’s certainly at least a plausible assumption.)
Assume: people with pre-existing medical conditions are more likely to be infected with COVID (at least in the ‘was noticed enough to be included in the study’ sense.)
Result: {COVID infection} correlates with {pre-existing medical conditions} correlates with {having medical checkups} correlates with {medical checkups cancelled or postponed} correlates with {issues not dealt with before they become a more major problem}
This can be controlled for to an extent—although given the ubiquity of cancellations in some places I’m not sure how large a sample remains after you do the control. Did they control for this? I don’t know. And them not controlling for this would be consistent with their conclusions, at least at a quick glance.
(There are likely other correlation paths.)
(Anecdote: I know multiple people with cancer right now that “should have” been caught earlier, but was missed until recently due to postponed or cancelled medical appointments.)