This post is a slight edit of a “how to read a paper” pdf by S. Keshav: We removed the last remaining redundancy from the original pdf, so read carefully for full benefit.

It has come to our attention that many folks read few papers, even if deeply; It is our observation that those who’ve read many papers shallowly as well understand the manifold of a research field much better than those who only read papers deeply. Depth is important for the key papers, but throughput is critical for seeing the broader map and learning to make use of low-quality papers to build intuition. This seems especially true in pre-paradigmatic fields such as alignment. This PDF is a nice little intro to reading lots of papers fast, and I find myself frequently wanting to reference it.

Edit to add note (gearstoascension):

If you read few papers, and this post does not influence your likelihood to read more papers, I’d love to hear a step by step trace-through of what repels you from the actions that discover papers and why you don’t read them—eg, are you busy? not expecting to understand? sure you already know better? etc. The goal here is to argue that those who are doing research should likely read dozens to hundreds of abstracts a month, first-pass tens of papers, and second-pass at least a few. If you aren’t doing that, and this post gets you nodding but doesn’t change your behavior, I’d be interested to know why.

Introduction: Up to three passes

Learning to efficiently read a paper is a critical but rarely taught skill. This two page pdf introduces a key idea that you should read the paper in up to three passes, instead of starting at the beginning and plowing your way to the end.

Each pass accomplishes specific goals and builds upon the previous pass: The first pass gives you a general idea about the paper. The second pass lets you grasp the paper’s content, but not its details. The third pass helps you understand the paper in depth.

(Zeroth pass: browse)

This isn’t mentioned in the linked pdf, but it’s also important to have high throughput on finding papers to read that have a shot at being both relevant and insightful, which can mean they’re outside your explicit field. More in a later post or comments. Paper discovery is a separate topic from paper reading, but one key strategy is browsing the citation graph, which is an incredibly powerful tool, especially if you don’t limit yourself to your field^[1].

First pass: skim

Get a bird’s eye view of the paper in 5-10 minutes. Read the title, abstract and introduction, and try to mentally predict what the conclusion might look like.

1. Carefully read the title, abstract, and introduction

2. Read section and subsection headings, but ignore everything else

3. Read conclusions

4. Glance over the references, mentally ticking off the ones you’ve already read

Often, figuring out what problem a paper solves is not obvious. By writing down 1-2 sentences of what that problem really is, you would force your brain to think of questions that might help reduce the problem to an essential puzzle that you might be able to tackle.

Goal for pass 1: become able to answer the five Cs:

1. Category: What type of paper is this—A measurement paper? An analysis of an existing system? A description of a research prototype?

2. Context: Which other papers is it related to? Which theoretical bases were used to analyze the problem?

3. Correctness: Do the assumptions appear to be valid?

4. Contributions: What are the paper’s main contributions?

5. Clarity: Is the paper well written?

Choose whether to read further. This could be because the paper doesn’t interest you, or you don’t know enough about the area to understand the paper^[2], or that the authors make invalid assumptions. The first pass is adequate for papers that aren’t in your research area, but may someday prove to be relevant. If you’re surprised by the paper’s contents after making your guess, and it’s topically relevant, it’s likely worth pass 2.

Second pass: read

Read the paper with greater care. Ignore details such as proofs. Make notes of key points (eg with note tools^[3]). Mark relevant unread references for further reading. This pass should take an hour or less. After this pass, you should be able to summarize the gist of the paper, with supporting evidence, to someone else.

Look carefully at the figures, diagrams and other illustrations in the paper. Pay special attention to graphs.

- Are axes properly labeled?
- Are results shown with error bars, so that conclusions are statistically significant?
- Can you identify correlation from causation?

Common mistakes like these will separate rushed, shoddy work from the truly excellent.
Mark relevant unread references for further reading^[4]. This is a good way to learn more about the background of the paper, and to create a mental map of the field.

Sometimes you still won’t understand, e.g. if the subject matter is new to you, has unfamiliar terminology or acronyms, or the authors use a proof or experimental technique that you don’t understand, so that the bulk of the paper is incomprehensible. The paper may be poorly written^[5] with unsubstantiated assertions and numerous forward references. Or it could just be that it’s late at night and you’re tired. You can now choose to:

(a) set the paper aside, hoping you don’t need to understand the material

(b) return to the paper later, perhaps after reading background material

Third pass: understand

The key to the third pass is to attempt to virtually (or if it’s simple enough, literally!) re-implement the paper: that is, making the same assumptions as the authors, re-create the work. You’d then think about how you would approach the problem, what roadblocks you see, and how the authors managed to solve those problems. By comparing this re-creation with the actual paper, you can easily identify not only a paper’s innovations, but also its hidden failings and assumptions.

This pass requires great attention to detail. Identify and challenge every assumption in every statement. This comparison of the actual with the virtual lends a sharp insight into the proof and presentation techniques in the paper and you can very likely add this paper to your repertoire of mental tools. During this pass, you should also jot down ideas for future work.

This pass can take about four or five hours for beginners, and about an hour for an experienced reader. At the end of this pass, you should be able to reconstruct the entire structure of the paper from memory^[7], as well as be able to identify its strong and weak points. In particular, you should be able to pinpoint implicit assumptions, missing citations to relevant work, and potential issues with experimental or analytical techniques.

The PDF is barely longer. We didn’t copy the following sections: Doing a literature survey; Keshav’s experience with this technique; Related work; A request by Keshav to make the pdf a living document; Acknowledgements; and References.

^
See other posts, especially:
- Tools for finding information on the internet: several very useful search tools, a couple of which are specific to papers and quite good at their jobs
- How to get into independent research on alignment/agency, especially the “reading” section
- the rest of the https://www.lesswrong.com/tag/scholarship-and-learning tag
-
^
though it may be worth reading papers in areas you don’t understand specifically because you do not understand them, before you have studied the basics in that field in depth. If you do so, simply keep in mind that you’re not expecting to fully understand the paper mechanistically as you attempt to do pass 2 and 3.
^
using a tool that makes taking notes easily is strongly recommended. If you don’t have one, consider zotero or hypothes.is; if you do have one and it’s neither of those, please share it in the comments, and what makes it a good choice for your workflow.
^
tools for marking and tracking papers can make this significantly easier; TGTA currently uses semanticscholar’s reader and citation browser when possible, and papermap.xyz to see the citation graph (which is a small site run by a single researcher and may go down if it gets popular suddenly, but is also very very nice to use. don’t put your secret 1934 capabilities-relevant math paper that nobody has remembered in decades into it, but it should be totally fine to use it to organize recent papers as normal researchers know about them anyway).
^
Note that machine learning papers are known for often being poorly written and difficult to read at a pass 2 or above level, even for very experienced researchers. Here’s a paper complaining about this in detail (or for an easier read in-browser, see ar5iv readable html5 version of the pdf), or here’s a reddit discussion post where folks express their difficulties with paper reading
^
Productive confusion is an interesting topic to search for itself! See also the noticing confusion tag.
^
In a famous quote by Feynman, he emphasizes the difference between knowing and understanding, sort of analogous to the difference b/w a second and a third pass.
See that bird? It’s a brown-throated thrush, but in Germany it’s called a Halzenfugel, and in Chinese they call it a Chung Ling and even if you know all those names for it, you still know nothing about the bird. You only know something about people; what they call the bird. Now that thrush sings, and teaches its young to fly, and flies so many miles away during the summer across the country, and nobody knows how it finds its way. You could learn the name of the thrush in all the languages, and you’d understand absolutely nothing about its behavior.

How to Read Papers Efficiently: Fast-then-Slow Three pass method