Everyone here is probably familiar with the reproducibility crisis in psychology and various other fields. I’ve recently been thinking there’s something very odd about all this. Namely, the reproducibility crisis seems to be almost entirely based on randomized controlled trials (RCTs). In terms of statistical design, these are the absolute gold-standard! Yet, my impression is that the majority of results in the social sciences are based on observational studies, not RCTs. While there’s usually at least some attempt to control for confounders, I feel like all the problems that contribute to the reproducibility crisis so far are 10x worse here: there’s so many more degrees of freedom in how you could set up the analysis.
Is my perception that the reproducibility crisis hasn’t really gotten to observational studies yet correct? If so, why not? And am I right to think that if/when these start getting checked, they are likely to be found even more unreliable?
I find it so puzzling that these seem to have mostly escaped scrutiny so far, and wonder if there’s a whole movement somewhere that I just haven’t encountered.
[Question] Why has the replication crisis affected RCT-studies but not observational studies?
Everyone here is probably familiar with the reproducibility crisis in psychology and various other fields. I’ve recently been thinking there’s something very odd about all this. Namely, the reproducibility crisis seems to be almost entirely based on randomized controlled trials (RCTs). In terms of statistical design, these are the absolute gold-standard! Yet, my impression is that the majority of results in the social sciences are based on observational studies, not RCTs. While there’s usually at least some attempt to control for confounders, I feel like all the problems that contribute to the reproducibility crisis so far are 10x worse here: there’s so many more degrees of freedom in how you could set up the analysis.
Is my perception that the reproducibility crisis hasn’t really gotten to observational studies yet correct? If so, why not? And am I right to think that if/when these start getting checked, they are likely to be found even more unreliable?
I find it so puzzling that these seem to have mostly escaped scrutiny so far, and wonder if there’s a whole movement somewhere that I just haven’t encountered.