Thanks! I once wrote up a somewhat-parallel discussion on a different topic in Section 5.1 here:
… So this is the “null hypothesis” of what to expect if there’s no such thing as [blah]. By now there are probably ≈1000 person-years of experimental data created by [blah] researchers. In such a huge mountain of data, there is bound to be lots of “random noise and ad hoc misinterpretations” that happen to line up remarkably with researchers’ prior expectations about [blah]. The question is not “Are there results that seems to provide evidence for [blah]?”, but rather “Is there much more evidence for [blah] than could plausibly be filtered out of 1000 person-years of random noise, misinterpretations, experimental errors, bias, occasional fraud, gross incompetence, weird equipment malfunctions, etc.?” …
and I also linked to & excerpted yet another parallel discussion on yet a different topic by Scott Alexander, Section 17 here.
Scott Alexander post that seems very relevant to your example: The Control Group Is Out Of Control. It puts into question even the heuristic of “Is there much more evidence for [blah] than...”.
Thanks! I once wrote up a somewhat-parallel discussion on a different topic in Section 5.1 here:
and I also linked to & excerpted yet another parallel discussion on yet a different topic by Scott Alexander, Section 17 here.
Scott Alexander post that seems very relevant to your example: The Control Group Is Out Of Control. It puts into question even the heuristic of “Is there much more evidence for [blah] than...”.