Here’s a prior that served me well for reading empirical literature:
1.) There is no effect (the null is true).
2.) If there is an effect but the data is observational, be very careful of any causal claims (they are most likely either due to modeling issues, bias due to confounding they missed, or getting the causal analysis wrong, or [a thousand more things]).
3.) If there is an effect and it is causal, I probably already heard about it, and there are lots of papers establishing it. Give the publication rate, and my reading rate, the chances of me stumbling on a genuinely new empirical result being reported for the first time is quite low.
4.) Conditional on me reading a paper, it’s either related to what I do, or the authors are “good at the media,” or (very rarely) it’s actually a breakthrough!
5.) Most papers are crap, most wrong findings are not retracted (incentives).
Here’s a prior that served me well for reading empirical literature:
1.) There is no effect (the null is true).
2.) If there is an effect but the data is observational, be very careful of any causal claims (they are most likely either due to modeling issues, bias due to confounding they missed, or getting the causal analysis wrong, or [a thousand more things]).
3.) If there is an effect and it is causal, I probably already heard about it, and there are lots of papers establishing it. Give the publication rate, and my reading rate, the chances of me stumbling on a genuinely new empirical result being reported for the first time is quite low.
4.) Conditional on me reading a paper, it’s either related to what I do, or the authors are “good at the media,” or (very rarely) it’s actually a breakthrough!
5.) Most papers are crap, most wrong findings are not retracted (incentives).