Step 1: Read the title, journal name, author list, and affiliations.
By reading papers in a field, talking to people in the field, and generally keeping track of the field as a social enterprise, you should be able to place papers in a context even before reading them. People absolutely have reputations, and that should inform your priors. You should also have an understanding of what the typical research methods are to answer a certain question—check either the title or the abstract to make sure that the methods used match the problem.
Actually, you know what?
Step 0: Spend years reading papers and keeping track of people to develop an understanding of trust and reputation as various results either pan our or don’t. Read a few textbooks to understand the physical basis of the commonly-used experimental and theoretical techniques, then check that understanding by reading more papers and keeping track of what kind of data quality is the standard in the field, how techniques are best applied, and which techniques and methods of analysis provide the most reliable results.
For example, by combining steps 0 and 1, you can understand that certain experimental techniques might be more difficult and easier to fool yourself with, but might be the best method available for answering some specific question. If you see a paper applying this technique to this sort of question, this actually should increase your confidence in the paper relative to the base rate for this technique, because it shows that the authors are exercising good judgment. Next...
Step 2: Read the abstract and look at the figures.
This is good for understanding the paper too, not just evaluating trustworthiness. Look for data quality (remember that you learned how to judge the data quality of the most common techniques in step 0) and whether they’ve presented it in a way that clearly backs up the core claims of the abstract, or presents the information you’re trying to learn from the paper. Data that is merely suggestive of the authors’ claims is actually a red flag, because remember, everyone just presents the nicest figure they can. Responsible scientists reduce their claims when the evidence is weak.
Step 3: Read the paper.
If you have specific parts you know you care about, you can usually just read those in detail and skim the rest. But if you really care about assessing this particular paper, check the procedures and compare it to your knowledge of how this sort of work should go. If there are specific parts that you want to check yourself, and you can do so, do so. This is also useful so you can...
Step 4: Compare it to similar papers.
You should have background knowledge, but it’s also useful to keep similar papers (both in terms of what methods they used, and what problem they studied) directly on hand if you want to check something. If you know a paper that did a similar thing, use that to check their methods. Find some papers on the same problem and cross-check how they present the details of the problem and the plausibility of various answers, to get a feel for the consensus. Speaking of consensus, if there are two similar papers from way in the past that you found via Google Scholar and one of them has 10x the citations of the other, take that into account. When you notice confusing statements, you can check those similar papers to see how they handled it. But once you’re really getting into the details, you’ll have to...
Step 5: Follow up citations for things you don’t understand or want to check.
If someone is using a confusing method or explanation, there should be a nearby citation. If not, that’s a red flag. Find the citation and check whether it supports the claim in the original paper (recursing if necessary). Accept that this will require lots of work and thinking, but hey, at least this feeds back into step 0 so you don’t have to do it as much next time.
Step 6: Ask a friend.
There are smart people out there. Hopefully you know some, so that if something seems surprising and difficult to understand, you can ask them what they think about it.
if there are two similar papers from way in the past that you found via Google Scholar and one of them has 10x the citations of the other, take that into account.
This seems great for figuring out the consensus in a field, but not for identifying when the consensus is wrong.
Here’s an answer for condensed matter physics:
Step 1: Read the title, journal name, author list, and affiliations.
By reading papers in a field, talking to people in the field, and generally keeping track of the field as a social enterprise, you should be able to place papers in a context even before reading them. People absolutely have reputations, and that should inform your priors. You should also have an understanding of what the typical research methods are to answer a certain question—check either the title or the abstract to make sure that the methods used match the problem.
Actually, you know what?
Step 0: Spend years reading papers and keeping track of people to develop an understanding of trust and reputation as various results either pan our or don’t. Read a few textbooks to understand the physical basis of the commonly-used experimental and theoretical techniques, then check that understanding by reading more papers and keeping track of what kind of data quality is the standard in the field, how techniques are best applied, and which techniques and methods of analysis provide the most reliable results.
For example, by combining steps 0 and 1, you can understand that certain experimental techniques might be more difficult and easier to fool yourself with, but might be the best method available for answering some specific question. If you see a paper applying this technique to this sort of question, this actually should increase your confidence in the paper relative to the base rate for this technique, because it shows that the authors are exercising good judgment. Next...
Step 2: Read the abstract and look at the figures.
This is good for understanding the paper too, not just evaluating trustworthiness. Look for data quality (remember that you learned how to judge the data quality of the most common techniques in step 0) and whether they’ve presented it in a way that clearly backs up the core claims of the abstract, or presents the information you’re trying to learn from the paper. Data that is merely suggestive of the authors’ claims is actually a red flag, because remember, everyone just presents the nicest figure they can. Responsible scientists reduce their claims when the evidence is weak.
Step 3: Read the paper.
If you have specific parts you know you care about, you can usually just read those in detail and skim the rest. But if you really care about assessing this particular paper, check the procedures and compare it to your knowledge of how this sort of work should go. If there are specific parts that you want to check yourself, and you can do so, do so. This is also useful so you can...
Step 4: Compare it to similar papers.
You should have background knowledge, but it’s also useful to keep similar papers (both in terms of what methods they used, and what problem they studied) directly on hand if you want to check something. If you know a paper that did a similar thing, use that to check their methods. Find some papers on the same problem and cross-check how they present the details of the problem and the plausibility of various answers, to get a feel for the consensus. Speaking of consensus, if there are two similar papers from way in the past that you found via Google Scholar and one of them has 10x the citations of the other, take that into account. When you notice confusing statements, you can check those similar papers to see how they handled it. But once you’re really getting into the details, you’ll have to...
Step 5: Follow up citations for things you don’t understand or want to check.
If someone is using a confusing method or explanation, there should be a nearby citation. If not, that’s a red flag. Find the citation and check whether it supports the claim in the original paper (recursing if necessary). Accept that this will require lots of work and thinking, but hey, at least this feeds back into step 0 so you don’t have to do it as much next time.
Step 6: Ask a friend.
There are smart people out there. Hopefully you know some, so that if something seems surprising and difficult to understand, you can ask them what they think about it.
This seems great for figuring out the consensus in a field, but not for identifying when the consensus is wrong.