Paper reading as a Cargo Cult

jem-mosig7 Aug 2022 7:50 UTC
70 points
10 comments5 min readLW link

I have come across various people (including my past self) who meet up regularly to study, e.g., alignment forum posts and discuss them. This helps people bond over their common believes, fears, and interests, which I think is good, but in no way is this ever going to lead anyone to find a solution to the alignment problem. In this post I’ll reason why this doesn’t help, and what I think we should do instead.

The cult

Reading good papers can be fun. You learn something interesting and, if the topic is hard but well presented by the authors, you get a kick from finally understanding something complicated. But is what you learned actually useful for the problem at hand? What is the question that drove you to read this paper in such detail?

Yes, you need to regularly skim papers for fun, so you get an idea of what’s out there and where to look when you need something. You also need to absorb terminology and good writing practice, so you can communicate your own research. Yet, I believe that fun-reading should only occupy a tiny fraction of your time, as you have more important things to do (see next section).

Despite its relative unimportance, paper reading groups tend to focus a lot on this fun-reading aspect. They are more of a social gathering than a mechanism to boost progress. Here is a typical pattern of Cargo Cult paper reading:

  1. People put papers on a list and vote on what they want to read next

  2. Everyone reads the paper that had the most votes (perhaps section by section)

  3. If somebody doesn’t understand something in the paper, (s)he asks the group and it is discussed until either everyone says that they understand, or it is agreed that this is just unclear or even wrong in the paper

  4. Repeat

Step 3 has some merit (if done right) but everything else is Cargo Cult. At best, it helps you stay up to date with the most mainstream papers in your field. But to do this, you don’t need to read all of these papers in detail.

The democratic paper selection progress is in fact the opposite of what you should do. Most of the time, it will promote whatever is most mainstream and, consequently, promote directions that most other people are already working on. This is a way to imitate research—not to do it. Hence, a Cargo Cult.

Actual science

To drive scientific progress means to do something that nobody else has ever done before[1]. This means that your idea or line of research tends to seem strange to others (at first sight). At the same time, it also tends to seem obvious to you—it’s just the natural next step when you take seriously what you’ve learnt so far.

Before I properly reconcile “strange” and “obvious” here, let me warn you of a trap: It is very easy to have an idea that seems obvious to you, but strange to others, when you are delusional. Especially when you are good at arguing, you can easily make yourself believe that you are right and everybody else is just not seeing it. Beware that trap.

The reason why your discovery/​invention/​insight will appear obvious to you is because it is a (short sequence of) small inferential step(s) from what you already know, and most of what you already know is widely accepted by the community. Ideally you also have more than one line of inference from common knowledge to your new discovery.

The reason why your discovery/​invention/​insight often appears strange to others is because you start reasoning from a particular subset of common knowledge. That is, while each fact that you start from is known by some people, only you know this particular combination of facts, and (critically!) you have dense knowledge about them[2].

Some examples:

  • Most of the mathematical laws that govern electro-magnetism where already widely known when Maxwell observed a slight inconsistency in them. When he fixed this with a small term, he made sure to not contradict any of the known experimental results. But the fix also allowed him to explain all of classical optics. Because Maxwell’s newly completed equations of electromagnetism describe a wave that propagates at the speed of light[3].

  • People had found fossiles of marine life on mountains, and knew that the shapes of continents fit roughly together. So various bright minds, from Da Vinci to Wegner gradually developed the idea of continental drift. The idea seemed very strange at the time, so the opposition was huge. But eventually, the idea prevailed, as it fit the growing evidence better than all alternatives.

  • The history of evolution shows the same patterns: Science is made up of gradual, small changes. Some of which sound more preposterous to contemporary people than others.

While all of this progress was made possible by building of other people’s insights, the actual step of progression comes from people pursuing odd perspectives and sticking to the facts. What we see as breakthroughs now are all gradual, small steps that didn’t follow the mainstream.

So instead of reading papers for fun, start at the beginning. What is it that you know right now. How would you frame the alignment problem? You probably don’t see a clear path from wherever you are to a solution (if you do, please tell!). But you can work towards it. If you don’t know the shape of the maze, you can at least try to walk towards the center where the trophy is kept.

The point is not that this way you will succeed. The point is that this way somebody has a chance to succeed. We cover a lot more territory when each of us works towards the goal from their current starting position, instead of most of us working towards the starting positions of a select few and then trying to continue from there.

If you don’t immediately see a direction that might be fruitful, think about it for five minutes. After that, if all directions that you can think of still seem equally improbable to lead to any useful result, pick a simpler problem and practice.

As my former supervisor liked to say: Where does the thing want to go? Play with it, and you’ll learn.

Caveats

Please keep gathering

Having said all this, I want to highlight that gathering and discussing alignment research is generally good. Not only are social gatherings essential for our mental health, but you also still have that third step in the paper reading process, where we help each other out to understand something. We may also inspire each other to creative thought, or tear down bad ideas.

Sometimes, you do need to read a lot

Sometimes you find that a whole branch of, e.g., mathematics exists, that seems very useful to what you are doing. For example, (if I remember correctly) Einstein realised that differential geometry was something he needed to understand, so he could formulate his intuitions mathematically. Hence, he learned it. Practiced and made it his own. In such situations it is very useful to work through a good book.

Note that this is different from fun-reading. Here, you already have a research direction of your own. This direction requires you to expand or solidify your knowledge on a specific subject, and somebody happened to have written a book/​paper about it.

This is not new

I think nearly everything that I’ve written here has appeared in one way or another in the Sequences. I just can’t find it and it might be useful for some to have it framed with the paper reading group phenomenon as an example.

  1. ^

    One could argue that I should add “and published it” here. But then dangerous things should not be published and you can also see things as “doing research for yourself”. I am avoiding this complication here, as it is essentially off topic.

  2. ^

    I use the term “fact” here because it is easy to visualise sets and subsets when you talk about concrete things like “facts”. But dense knowledge is not a collection of facts.

  3. ^

    Technically, you have to apply the curl operator to transform a subset of the equations into wave form. Doing this was way harder back then, because vector notation and curl operators had not been invented yet.

jem-mosig7 Aug 2022 7:50 UTC
70 points
10 comments5 min readLW link