arbitrarily decide that everything that predicts observations that would be only 5% likely if it was false is true and everything without such observations is false, regardless of how many observations were actually made
This was hard to parse. I would have named “p-value” directly. My understanding is that a stated “p-value” will indeed depend on the number of observations, and that in practice meta-analyses pool the observations from many experiments. I agree that we should not use a hard p-value cutoff for publishing experimental results.
I should have said “a set of observations” and “sets of observations”. I meant things like that if you and other groups test lots of slightly different bogus hypotheses 5% of them will be “confirmed” with statistically significant relations.
Got it, and agreed. This is one of the most pernicious forms of dishonesty by professional researchers (lying about how many hypotheses were generated), and is far more common than merely faking everything.
Good comment.
However,
This was hard to parse. I would have named “p-value” directly. My understanding is that a stated “p-value” will indeed depend on the number of observations, and that in practice meta-analyses pool the observations from many experiments. I agree that we should not use a hard p-value cutoff for publishing experimental results.
I should have said “a set of observations” and “sets of observations”. I meant things like that if you and other groups test lots of slightly different bogus hypotheses 5% of them will be “confirmed” with statistically significant relations.
Got it, and agreed. This is one of the most pernicious forms of dishonesty by professional researchers (lying about how many hypotheses were generated), and is far more common than merely faking everything.