Given that the replication rate of findings isn’t zero, it seems that some researchers aren’t completely fraudulent and at least partly “real work”.
Unfortunately, it’s not that easy. Even a stopped clock is right twice a day: there is a base rate of being ‘right by accident’. Meehl talks a lot about this: if you are in particle physics and you’re predicting the 15th digit of some constant, then the base rate is ~0% and so it’s hard to be right by accident; if you’re in psychology, you’re usually predicting whether some number is less than or bigger than zero (because it’s never zero) because most predictions are so ill-defined that they could be just about any size and no one will blink an eye, and so your base rate of being ‘right’ by accident is… substantially higher than 0%. So depending on you define ‘replication’, it’d be easy to have anywhere up to 50% replication rates with completely fraudulent fields of research.
Given your past writing I’m a bit surprised by that position. I thought you wrote that in a lot of cases, the causative effect of most interventions is very little.
The effect is of course not zero but I would expect them most of the time not to have effect sizes that are strong enough to show up.
If publishable true effects were that easy to come by, fraud wouldn’t be really needed.
The effect is of course not zero but I would expect them most of the time not to have effect sizes that are strong enough to show up.
I’m assuming that the replications have been sufficiently powered as to always exclude the null—if only because they are reaching the ‘crud factor’ level.
If publishable true effects were that easy to come by, fraud wouldn’t be really needed.
They aren’t easy to come by, which is why Many Labs and the replication efforts exist pretty much solely because Arnold chose to bankroll them. The history of meta-science and psychology in the 2010s would look rather different if one Texan billionaire had different interests.
Given that the replication rate of findings isn’t zero, it seems that some researchers aren’t completely fraudulent and at least partly “real work”.
An interesting question is how many failed replications are due to fraud. Are 20%? 50% or 80% of the studies that don’t replicate fraudulent?
Unfortunately, it’s not that easy. Even a stopped clock is right twice a day: there is a base rate of being ‘right by accident’. Meehl talks a lot about this: if you are in particle physics and you’re predicting the 15th digit of some constant, then the base rate is ~0% and so it’s hard to be right by accident; if you’re in psychology, you’re usually predicting whether some number is less than or bigger than zero (because it’s never zero) because most predictions are so ill-defined that they could be just about any size and no one will blink an eye, and so your base rate of being ‘right’ by accident is… substantially higher than 0%. So depending on you define ‘replication’, it’d be easy to have anywhere up to 50% replication rates with completely fraudulent fields of research.
Given your past writing I’m a bit surprised by that position. I thought you wrote that in a lot of cases, the causative effect of most interventions is very little.
The effect is of course not zero but I would expect them most of the time not to have effect sizes that are strong enough to show up.
If publishable true effects were that easy to come by, fraud wouldn’t be really needed.
I’m assuming that the replications have been sufficiently powered as to always exclude the null—if only because they are reaching the ‘crud factor’ level.
They aren’t easy to come by, which is why Many Labs and the replication efforts exist pretty much solely because Arnold chose to bankroll them. The history of meta-science and psychology in the 2010s would look rather different if one Texan billionaire had different interests.