A fundamental problem seems to be that there is a lower prior for any given hypothesis, driven by the increased number of researchers, use of automation, and incentive to go hypothesis-fishing.
Wouldn’t a more direct solution be to simply increase the significance threshold required in the field?
A fundamental problem seems to be that there is a lower prior for any given hypothesis, driven by the increased number of researchers, use of automation, and incentive to go hypothesis-fishing.
That doesn’t lower the pre-study prior for hypotheses, it (in combination with reporting bias) reduces the likelihood ratio a reported study gives you for the reported hypothesis.
Wouldn’t a more direct solution be to simply increase the significance threshold required in the field?
Increasing the significance threshold would mean that adequately-powered honest studies would be much more expensive, but those willing to use questionable research practices could instead up the ante and use the QRPs more aggressively. That could actually make the published research literature worse.
That doesn’t lower the pre-study prior for hypotheses, it (in combination with reporting bias) reduces the likelihood ratio a reported study gives you for the reported hypothesis.
Respectfully disagree. The ability to cheaply test hypotheses allows researchers to be less discriminating. They can check a correlation on a whim. Or just check every possible combination of parameters simply because they can. And they do.
That is very different from selecting a hypothesis out of the space of all possible hypotheses because it’s an intuitive extension of some mental model. And I think it absolutely reduces the pre-study priors for hypotheses, which impacts the output signal even if no QRPs are used.
A fundamental problem seems to be that there is a lower prior for any given hypothesis, driven by the increased number of researchers, use of automation, and incentive to go hypothesis-fishing.
Wouldn’t a more direct solution be to simply increase the significance threshold required in the field?
That doesn’t lower the pre-study prior for hypotheses, it (in combination with reporting bias) reduces the likelihood ratio a reported study gives you for the reported hypothesis.
Increasing the significance threshold would mean that adequately-powered honest studies would be much more expensive, but those willing to use questionable research practices could instead up the ante and use the QRPs more aggressively. That could actually make the published research literature worse.
Respectfully disagree. The ability to cheaply test hypotheses allows researchers to be less discriminating. They can check a correlation on a whim. Or just check every possible combination of parameters simply because they can. And they do.
That is very different from selecting a hypothesis out of the space of all possible hypotheses because it’s an intuitive extension of some mental model. And I think it absolutely reduces the pre-study priors for hypotheses, which impacts the output signal even if no QRPs are used.