A few months ago I read this paper for a class (paywalled). In it, the authors perform a similar set of knockdown experiments using both short hairpin RNA and CRISPR in order to repress a gene. Their results with the shRNAs are quite impressive, but the CRISPR part is less so. Why the disparity?
The key to this sort of thing is to picture it from the authors’ perspective. Read between the lines, and picture what the authors actually do on a day-to-day basis. How did they decide exactly which experiments to do, which analyses to run, what to write up in the paper?
In the case of the paper I linked above, the authors had a great deal of experience and expertise with shRNAs, but seemed to be new to CRISPR. Most likely, they tried out the new technique either because someone in the lab wanted to try it or because a reviewer suggested it. But they didn’t have much expertise with CRISPR, so they had some probably-spurious results in that part of the paper. All we see in the paper itself is a few results which don’t quite line up with everything else, but it’s not hard to guess what’s going on if we think about what the authors actually did.
This principle generalizes. The main things to ask when evaluating a paper’s reliability are things like:
Does it seem like the authors ran the numbers on every little subset of their data until they found p < .05?
Does it seem like the authors massaged the data until it gave the answer they wanted?
Does it seem like the authors actively looked for alternative hypotheses/interpretations of their data, and tried to rule them out?
… and so forth. In short, try to picture the authors’ actual decision-making process, and then ask whether that decision-making process will yield reliable results.
There’s all sorts of math you can run and red flags to watch out for—multiple tests, bad incentives, data not actually matching claims, etc—but at the end of the day, those are mostly just concrete techniques for operationalizing the question “what decision-making process did these authors actually use?” Start with that question, and the rest will follow naturally.
A few months ago I read this paper for a class (paywalled). In it, the authors perform a similar set of knockdown experiments using both short hairpin RNA and CRISPR in order to repress a gene. Their results with the shRNAs are quite impressive, but the CRISPR part is less so. Why the disparity?
The key to this sort of thing is to picture it from the authors’ perspective. Read between the lines, and picture what the authors actually do on a day-to-day basis. How did they decide exactly which experiments to do, which analyses to run, what to write up in the paper?
In the case of the paper I linked above, the authors had a great deal of experience and expertise with shRNAs, but seemed to be new to CRISPR. Most likely, they tried out the new technique either because someone in the lab wanted to try it or because a reviewer suggested it. But they didn’t have much expertise with CRISPR, so they had some probably-spurious results in that part of the paper. All we see in the paper itself is a few results which don’t quite line up with everything else, but it’s not hard to guess what’s going on if we think about what the authors actually did.
This principle generalizes. The main things to ask when evaluating a paper’s reliability are things like:
Does it seem like the authors ran the numbers on every little subset of their data until they found p < .05?
Does it seem like the authors massaged the data until it gave the answer they wanted?
Does it seem like the authors actively looked for alternative hypotheses/interpretations of their data, and tried to rule them out?
… and so forth. In short, try to picture the authors’ actual decision-making process, and then ask whether that decision-making process will yield reliable results.
There’s all sorts of math you can run and red flags to watch out for—multiple tests, bad incentives, data not actually matching claims, etc—but at the end of the day, those are mostly just concrete techniques for operationalizing the question “what decision-making process did these authors actually use?” Start with that question, and the rest will follow naturally.