I’d add to the basic statistical problem the vast overgeneralization and bad decision theory.
You hit on one part of that, the generalization to the entire population.
People are different.
But even if they’re the same, U shaped response curves make it unlikely to find a signal. - you have to have the goldilocks amount to show an improvement. People vary over time. going in and out of the goldilocks range. So you when you add something, you’ll be pushing some people into the goldilocks range, and some people out.
It also comes from multiple paths to the same disease. A disease is a set of observable symptoms, not the varying particular causes of the same symptoms. Of course it’s hard to find the signal in a batch of people clustered into a dozen different underlying causes for the same symptoms.
But the bad decisions theory is the worst part, IMO. If you have a chronic problem, a 5% chance of a cure from a low risk, low cost intervention is great. But getting a 5% signal out of black box testing regimes biased against false positives is extremely unlikely, and the bias against interventions that “don’t work” keeps many doctors from trying perfectly safe treatments that have a reasonable chance of working.
The whole outlook is bad. It shouldn’t be “find me a proven cure that works for everyone”. It should be “find me interventions to control the system in a known way.” Get me knobs to turn, and let’s see if any of the knobs work for you.
I’d add to the basic statistical problem the vast overgeneralization and bad decision theory.
You hit on one part of that, the generalization to the entire population.
People are different.
But even if they’re the same, U shaped response curves make it unlikely to find a signal. - you have to have the goldilocks amount to show an improvement. People vary over time. going in and out of the goldilocks range. So you when you add something, you’ll be pushing some people into the goldilocks range, and some people out.
It also comes from multiple paths to the same disease. A disease is a set of observable symptoms, not the varying particular causes of the same symptoms. Of course it’s hard to find the signal in a batch of people clustered into a dozen different underlying causes for the same symptoms.
But the bad decisions theory is the worst part, IMO. If you have a chronic problem, a 5% chance of a cure from a low risk, low cost intervention is great. But getting a 5% signal out of black box testing regimes biased against false positives is extremely unlikely, and the bias against interventions that “don’t work” keeps many doctors from trying perfectly safe treatments that have a reasonable chance of working.
The whole outlook is bad. It shouldn’t be “find me a proven cure that works for everyone”. It should be “find me interventions to control the system in a known way.” Get me knobs to turn, and let’s see if any of the knobs work for you.