This is cross-posted from my personal blog, where I share thoughts on my work and learning process in the OpenAI Scholars Program. I thank Ruby Bloom for suggesting that I share the post here as well.

OpenAI Scholars: Fifth Steps—The Dreaded Literature Review

The OpenAI Scholars , and I among them, recently completed a project proposal for the second half of our program. Having recently finished a PhD, writing a proposal and doing the requisite literature review, should be second nature. But literature reviews were always my least favorite part of research.

Fortunately, I’m in good company. In a previous life as a young aspiring neuroscientist, I once attended a talk by David Hubel . When asked for tips about how to “keep up with the literature”, I recall that Professor Hubel responded: “You know, at some point in your career, you have to decide if you want to be a consumer—or a producer.” He elaborated that he would only really look at papers that his advisor or his long-term collaborator Torsten Wiesel pointed him to.

There’s good reason to want to avoid literature reviews. To begin with, the problem formulation is intractable: “Know everything.” If you’ve spent any amount of time around academics, it will soon become apparent that this is exactly what they expect from you. “Oh you haven’t read that paper?” The assumption is that you have read every paper there is.

Of course, this is an impossible task. Last time I visited Google Scholar and searched on a few project-relevant terms, I encountered 105,000 papers. It takes me a full day to read and satisfactorily understand an academic paper. So reading 105 kilo-papers would take me 288 years. And I had hoped to also do some coding work during the Scholars program.

Now suppose you want to cook a ghormeh sabzi . To do so, you buy yourself a can of ghormeh sabzi mix , and follow the instructions on the can. There are about five steps. However, as ubiquitous an activity as the literature review is (in academic circles anyway), I am yet to encounter an honest and pragmatic recipe for how they are to be cooked. I really wish there was one.

I for one am a great fan of Thich Nhat Hanh’s how-to books. Some of my favorite titles among Nhat Hanh’s books are: “How to sit”, “How to relax”, and—perhaps the best one—“How to see”.

I wish there had been a series of books like that available for me as a beginning grad student, covering seemingly trivial but often intractable activities like “How to read a paper”, “How to attend a talk” and, indeed, “How to do a literature review”.

(**Updated: Apparently there exist plenty of expositions on this topic. See the comment by AllAmericanBreakfast below. I hope that some of the ideas in this post might nonetheless add something to whatever else you might have read.)

So, this is a first draft toward such a pragmatic guide to literature reviews: How to do them and how to avoid them. Again, this is absolutely in draft form, so please do send your constructive feedback my way.

How to do a literature review

1. Start with your goal.

Do not start your literature review with the goal of “doing a literature review”. Notice that such a goal does not have an end point when you can look at your work in satisfaction and notice that you are done. (Unless you have 288 years to spare). Instead, start with identifying your goal: Why are you doing this literature review? In my case, I had completed a spreadsheet with 50 papers, including their main points, type of neural recording, type of network architecture, a usefulness rating etc., by the time I realized that I had not identified my goal. I had to take a deliberate step back to recall that one goal of my literature review was to write a project proposal. Hence, my literature review should be tailored to that goal. Once I realized that, I decided to approach my work from the opposite direction: I drafted a project proposal “blind” to the literature. In other words, I wrote out a draft in bullet points with only the information I had in my mind. Once I had done that, it became quite obvious where I had knowledge gaps. Those knowledge gaps created the questions that I needed to search for in my literature review (see point 3 below.)

2. Decide what constitutes success: How will you know that you’re done?

This is really important, and it’s often the hardest part with an infinity-project like a literature review. In my case, the done-point was when I had completed the project proposal. But how do I know that my proposal is good enough? In practice, for me, it was when the deadline hit. If you have come up with better metrics and markers for when to consider a project completed, please do let me know.

3. Define your questions.

When doing a literature review, it’s critical to know what information you’re searching for. Trying to absorb all the information contained in a dense document is rarely a good plan. In my case, my questions fell out of my first, naive project proposal draft. I also talked my project through with a friend , who is a more experienced researcher. He offered the following thought points for guidance to a literature review for a new project:

1. The goal for your literature review should be to identify the problem that your project will solve. Your project might either aim to solve a new problem, or improve on an existing method.
2. Use the literature revew to identify *gaps* in the literature: What do we not yet know?
3. What are the shortcomings of existing methods to solve the problem? Why are current approaches not yet useful in practice? What is the bottleneck?

4. Answer your questions.

I think this is self-explanatory, but please let me know if not! One thing to keep in mind here is to not get lost in literature-tangents while finding answers to your questions. Provided your goal is to finish, that is. If you have the time to go off on philosophical tangents, feel free and enjoy!

5. Stop.

Just stop. Seriously. When you’re done, you should stop. Go take a walk outside.

How to avoid doing literature reviews

The first section described the second-best way of doing a literature review. The best way to do a literature review is to not do a literature review. The art of avoiding doing a literature review is very similar to the art of avoiding deep cleaning your house: Keep it tidy on a daily basis. (Just like with deep cleaning your house, you probably still need to do a proper deep dive into the literature from time to time, just not as frequently as you would if you didn’t have a routine of daily up-keep). Here are some ideas for “keeping up with the literature”:

1. Mind whom you follow on Google Scholar.

Once you’ve gotten your feet a little bit wet in your field of interest, you will have a clue of who often writes interesting papers. If you follow them on Google Scholar, you will be notified as they publish more interesting stuff.

2. Mind whom you follow on Twitter.

This is especially true of deep learning, but it’s becoming increasingly common in every scientific field: Those same people that you follow on Google Scholar—they will also be tweeting when they publish new cool stuff. Also when their friends publish new cool stuff. They might even highlight the most interesting aspect of their work through a figure, or a digestible blog post. Notice that whom you do *not* follow on Twitter is at least as important as whom you do follow. Optimize your twitter feed, so that you don’t clutter away an excellent opportunity to find out about the most exciting and inspiring new developments in your field.

3. Organize the papers that come flying your way.

To harness your daily literature upkeep for avoiding future literature reviews, be sure to organize the work you encounter. Mendeley is a useful, and mostly free, tool for doing this, but there are many of them. (Those of us who are old and remember typing out references letter by letter, will especially appreciate the convenience of a citation manager. It will never be a good use of your time to have to think about what part of a citation should be italicized.) Once the time comes to write your next proposal or paper, you have a treasure trove of results that you’ve already thought about, readily available at your fingertips.

4. Make it fun.

This is the most important point. It’s much easier to absorb information that you care about. One way that I hacked this in my current literature review is that I reached out to a number of authors that I had encountered in my literature search for an informal chat. This was one of the best experiences of scientific exchange I’ve ever had! I talked to scientists in Louisiana, Greece, and Australia, all of whom had several years more experience with my topic of interest than I did. Once you’ve heard a scientist tell you about their own paper—what part of the project was hard, what was interesting, what parts are only there because of reviewer request etc. - the content of their paper pops out at you in a much more colorful and living way. It’s no longer such a chore to look at a paper when, instead of some dry black-and-white characters on a page, it’s a friend’s documentation of their work and thought process. That said, it’s definitely a good idea to take a look at a paper *before* you reach out to its author. But having a scheduled call is also a good incentive to engage more deeply with your reading process.

There. I hope that’s somewhat helpful. Go forth, and enjoy your reading!

How to (not) do a literature review