I typed up the below message before discovering that the term I was looking for is “data dredging” or “hypothesis fishing.” Still decided to post below so others know.
Is there a well-known term for the kind of error that pre-registrations of scientific studies is meant to avoid? I mean the error where an experiment is designed to test something like “This drug cures the common cold,” but then when the results show no effect, the researchers repeatedly do the analysis on smaller slices of the data from the experiment, until eventually they have the results “This drug cures the common cold in males aged 40-60, p<.05,” when of course that result is just due to random chance (because if you do the statistical tests on 20 subsets of the data, chances are one of them will show an effect with p<.05).
It’s similar to the file drawer effect, except it’s within a single experiment, not many.
I typed up the below message before discovering that the term I was looking for is “data dredging” or “hypothesis fishing.” Still decided to post below so others know.
Publication bias