The LessWrong Annual Review is a central mechanism for reviewing the site’s content, it’s kind of like our site’s peer review process.
LessWrong has the goal of making intellectual progress on important problems. To make progress, you gotta examine your community’s outputs not only when they’re first published, but also once enough time has passed to see whether they continued to provide value after initial hype fades and flaws have had time to surface.
So far, the bulk of the LessWrong Review happens in the winter season, over December and January. Batched yearly, we review the posts from the previous year. 2019′s post get reviewed at the end of 2020 (i.e. every post in each review batch is 1-2 years old).
For two months, the community nominates, writes reviews, and votes on the most valuable posts. Reviewers ask questions like:
Does this post continue to affect my thinking or actions even 12+ months later?
Do its claims still seem valuable or interesting even now?
Was it invalidated by further work or other criticisms that came up?
Which follow-up work do I still wish someone would do?
The community answering these questions leads to benefits like:
We create broad knowledge about which ideas and results turned out to be robust and future work can be built upon (and which ideas failed to achieve that).
Out of the thousands of posts published each year, we can direct the attention of future readers to the most valuable content.
e.g. via the creation of Sequences, [e]books, etc.
We can better reward and incentivize the thinkers whose contributions were most valuable.
You can see the top posts from previous years on the Best of LessWrong page. You can see the individual results page for each Annual Review so far: 2018, 2019, 2020, 2021.
What is the Review process?
[New!] The LessWrong team is currently experimenting with ways to give the Annual Review year-long salience, i.e. by giving people the ability to bet in prediction markets about which posts will be rated highly in the Review, and by giving Review “winners” prominent treatment (e.g. cool art on their post pages).
1. A couple of weeks of nominating posts for consideration in the review.
2. A month of writing reviews and critiques of posts.
3. Two final weeks voting on which posts were ultimately most valuable.
The exact process of the Annual Review is not locked in stone and the LessWrong team might decide to change it dramatically in the future. That said, for the past five years it has followed approximately the same basic format:
Preliminary Voting Phase (2 weeks): We identify posts especially worthy of consideration in the review casting preliminary votes. Posts with 2 preliminary votes move into the Discussion Phase.
Discussion Phase (4 weeks): We review and debate posts. Posts that receive at least one written review move to the final voting phase.
Final Voting (2 weeks): We do a full voting pass, using quadratic voting. The outcome determines the Annual Review results.
Here’s a table of all the posts by year that were part of this process.