A single experiment, especially if it has a small sample size—but even large sample sizes can be ruined by experimental error etc.---or even a small number of such experiments which confirm each other—just aren’t going to give very reliable results for a complex system.
When things are tightly controlled in a lab for a simple system (with few variables), then basic statistical methods can yield believable results. This is why the replication problem is mostly in psychology, medicine, and social sciences (I presume biology & ecology should be on the list, but they aren’t so consequential maybe, except for the biology that is already in the medical category). Those disciplines work with complex systems that we have little macro-level understanding of and do not yield themselves to be controlled well in a lab. Whatever lab controls can be instituted really control very little because we are often talking about human persons who are complex biological systems themselves.
Of course the other problem is that the researchers themselves generally do not understand statistical theory. And that is a very big factor here generally. You basically have to be an actual statistician or very quantitatively/analytically skilled. There is statistical literature, for example, about the problems with p-values and statisticians trying to figure out what to do about that since nobody understands these things.
All that being said, with many iterations of experiments addressing the same question, say, done by many different research teams (say in different places and at different times, to whatever appropriate level), then we can start to see real physical connections emerge. it might still be the case that some hidden confounding variable is the ultimate arbiter of whatever relationship is observed, but the observed relationship will still be real but just mediated but unknown observables. It would be better to know the precise causal chain, but it’s still pretty cool to know a real relationship still, even if it is only correlation and not causation.
It costs a lot of money to do so many repetitions. So for something that doesn’t ultimately have much economic impact, there is no incentive to spend a lot of money to get some small bits of information to be cataloged away and rarely if ever accessed. Even if large sums of money are spent to answer some question about a complex system, the outcome might still be uncertainty about whether the relationship observed is real.
A single experiment, especially if it has a small sample size—but even large sample sizes can be ruined by experimental error etc.---or even a small number of such experiments which confirm each other—just aren’t going to give very reliable results for a complex system.
When things are tightly controlled in a lab for a simple system (with few variables), then basic statistical methods can yield believable results. This is why the replication problem is mostly in psychology, medicine, and social sciences (I presume biology & ecology should be on the list, but they aren’t so consequential maybe, except for the biology that is already in the medical category). Those disciplines work with complex systems that we have little macro-level understanding of and do not yield themselves to be controlled well in a lab. Whatever lab controls can be instituted really control very little because we are often talking about human persons who are complex biological systems themselves.
Of course the other problem is that the researchers themselves generally do not understand statistical theory. And that is a very big factor here generally. You basically have to be an actual statistician or very quantitatively/analytically skilled. There is statistical literature, for example, about the problems with p-values and statisticians trying to figure out what to do about that since nobody understands these things.
All that being said, with many iterations of experiments addressing the same question, say, done by many different research teams (say in different places and at different times, to whatever appropriate level), then we can start to see real physical connections emerge. it might still be the case that some hidden confounding variable is the ultimate arbiter of whatever relationship is observed, but the observed relationship will still be real but just mediated but unknown observables. It would be better to know the precise causal chain, but it’s still pretty cool to know a real relationship still, even if it is only correlation and not causation.
It costs a lot of money to do so many repetitions. So for something that doesn’t ultimately have much economic impact, there is no incentive to spend a lot of money to get some small bits of information to be cataloged away and rarely if ever accessed. Even if large sums of money are spent to answer some question about a complex system, the outcome might still be uncertainty about whether the relationship observed is real.