Here’s a structural explanation for statistical improprieties in scientific research:
For publications, scientists are incentivized to overinvest in precision. They aren’t able to get a publication out of just their informal observations and qualitative data.
Yet scientists are also underincentivized to invest in definitive answers. That doesn’t mean there’s no incentive, just that the incentives aren’t adequate.
“Do precise statistics, but it’s not that important to conclusively resolve the question at hand” sounds like a recipe for wasted effort. Once you realize that your research isn’t definitive, don’t know how to make it definitive, and that nobody’s expecting you to achieve this anyway, it’s not that far to go to start neglecting the statistics as well.
A better recipe would be “gather the data that gives a definitive answer to your real question, and demonstrate that your conclusions are sound using whatever statistics or demonstrations make the most sense.”
We can also think about a funnel model. You should start with broad and informal (cheap) qualitative observation, and then progressively narrow down to formal and precise quantitative data as you figure out what the few key most important variables are.
Here’s a structural explanation for statistical improprieties in scientific research:
For publications, scientists are incentivized to overinvest in precision. They aren’t able to get a publication out of just their informal observations and qualitative data.
Yet scientists are also underincentivized to invest in definitive answers. That doesn’t mean there’s no incentive, just that the incentives aren’t adequate.
“Do precise statistics, but it’s not that important to conclusively resolve the question at hand” sounds like a recipe for wasted effort. Once you realize that your research isn’t definitive, don’t know how to make it definitive, and that nobody’s expecting you to achieve this anyway, it’s not that far to go to start neglecting the statistics as well.
A better recipe would be “gather the data that gives a definitive answer to your real question, and demonstrate that your conclusions are sound using whatever statistics or demonstrations make the most sense.”
We can also think about a funnel model. You should start with broad and informal (cheap) qualitative observation, and then progressively narrow down to formal and precise quantitative data as you figure out what the few key most important variables are.