Already partially mentioned by others, including OP.
I usually start with comparing the conclusion with my expectations (I’m painfully aware that this creates a confirmation bias, but what else am I supposed to compare it with). If they are sufficiently different I try to imagine how, using the method described by the authors, I would be able to get a positive result to their experiment conditional on my priors being true, i.e. their conclusion being false. This is basically the same as trying to figure out how I would run the experiment and which data would disprove my assumptions, and then seeing if the published results fall in that category.
Usually the buck stops there, most published research use methods that are sufficiently flimsy that (again, conditional on my priors), it is very likely the result was a fluke. This approach is pretty much the same as your third bullet point, and also waveman’s point number 5. I would like to stress though that it’s almost never enough to have a checklist of “common flaws in method sections” (although again, you have to start somewhere). Unfortunately different strengths and types of results in different fields require different methods.
A small Bayesian twist on the interpretation of this approach: when you’re handed a paper (that doesn’t match your expectations), that is evidence of something. I’m specifically looking at the chance that, conditional on my priors being accurate, the paper I’m given is still being published.
Already partially mentioned by others, including OP.
I usually start with comparing the conclusion with my expectations (I’m painfully aware that this creates a confirmation bias, but what else am I supposed to compare it with). If they are sufficiently different I try to imagine how, using the method described by the authors, I would be able to get a positive result to their experiment conditional on my priors being true, i.e. their conclusion being false. This is basically the same as trying to figure out how I would run the experiment and which data would disprove my assumptions, and then seeing if the published results fall in that category.
Usually the buck stops there, most published research use methods that are sufficiently flimsy that (again, conditional on my priors), it is very likely the result was a fluke. This approach is pretty much the same as your third bullet point, and also waveman’s point number 5. I would like to stress though that it’s almost never enough to have a checklist of “common flaws in method sections” (although again, you have to start somewhere). Unfortunately different strengths and types of results in different fields require different methods.
A small Bayesian twist on the interpretation of this approach: when you’re handed a paper (that doesn’t match your expectations), that is evidence of something. I’m specifically looking at the chance that, conditional on my priors being accurate, the paper I’m given is still being published.