The 2020 Review
Today is the first day of the LessWrong 2020 Review. At the end of each year, we take a look at the best posts from the previous year, and reflect on which of them have stood the test of time.
As we navigate the 21st century, a key issue is that we’re pretty confused about which intellectual work is valuable. We have very sparse reward signals when it comes to “did we actually figure out things that matter?”
The Review has a few goals. It improves our incentives, feedback, and rewards for contributing to LessWrong. It creates common knowledge about the LW community’s collective epistemic state about the most important posts of 2020. And it turns all of that into a highly curated sequence that people can read. You can read more about the philosophy behind the Review in last year’s announcement post.
A few important announcements about this year’s review:
We’ve replaced the nomination process with Preliminary Voting.
Winning posts will get Donation Buttons.
There’s a new View My Past Upvotes page to help you find posts to vote on.
2019 Books will be shipping in a couple weeks. We’re still evaluating whether and how to do books for 2020.
How does the review work?
The review has three phases:
Preliminary Voting Phase (Dec 1- 14)
Discussion Phase (Dec 14 - Jan 11)
Final Voting (Jan 11 to Jan 25)
Users who registered before January 1st 2020 can vote. The LessWrong moderation team will take the results of the vote as input for a curated sequence of posts, and award prizes. We’ll be giving more weight to the votes of users with 1000+ karma.
Preliminary Voting
The first big change this year is changing the Nomination Phase to the Preliminary Vote Phase. Eligible voters voters will see this UI:
If you think a post was an important intellectual contribution, you can cast a vote indicating roughly how important it was. A vote of 1 means “it was good.” A vote of 4 means “it was quite important”, and is weighted 4x a vote of 1. A vote of 9x means it was a crucial piece of intellectual progress.
You can vote at the top of a post, or anywhere the post appears in a list (like the All Posts page, or the new View Your Past Upvotes page).
Posts that get at least one positive vote go to the Voting Dashboard, where other users can vote on it. You’re encouraged to give at least a rough vote based on what you remember from last year.
If you feel a post was important, you’re also encouraged to write up at least a short review of it saying what stands out about the post and why it matters. (This is essentially the same as writing a nomination comment from the 2018 and 2019 Reviews. In practice nominations and reviews were fairly similar and it didn’t seem worth separating them out in the UI). You’re allowed to write multiple reviews of a post, if you want to start by jotting down your quick impressions, and later review it in more detail.
Why did we switch to preliminary voting?
Each year, more posts get written on LessWrong. The first Review of 2018 considered 1,500 posts. In 2020, there were 3,000. Processing that many posts is a lot of work.
Preliminary voting is designed to help handle the increased number of posts. Instead of simply nominating posts, we start directly with a votes. At the end of the Preliminary Voting phase, the results of the vote will be published. This will help the LessWrong community prioritize reviews. Posts that are highly ranked can invite more investigation of how they stand the tests of time. If you think a post was (unfairly) ranked low, you are welcome to write a positive review arguing it should be considered more strongly.
Posts which everyone agrees are “meh” can get deprioritized, making more time for more interesting posts.
How is preliminary voting calculated?
You can cast an unlimited number of votes. However, the more votes you cast, and the higher your total “score” (where a “9” vote counts for 9x the score of a “1” vote), the less influential each of your votes will be. We normalize voting strength so that all users who are past a certain “score” threshold exert roughly the same amount of total influence.
On the back end, we use a modified quadratic voting system, which allocates a fixed number of “points” across your votes based on how strong they are.
Final Voting
Posts that receive at least one review move on the Final Voting Phase. The UI will require voters to at least briefly skim reviews before finalizing their vote for each post, so arguments about each post can be considered.
As with last year, we’ll publish the voting results for users with 1000+ karma, as well as all users. The LessWrong moderation team will take the voting results as a strong indicator of which posts to include in the Best of 2020 sequence.
(Note: I am currently uncertain whether Final Voting will use the fine-tuned quadratic system from last year. I plan to take last year’s voting data, round each vote to the nearest “1, 4, or 9”, and see if the results are significantly different from the original vote. If they aren’t very different, I suspect it may not make sense to encourage everyone to spend a bunch of time fine-tuning their quadratic points. I’m open to arguments in either direction)
Donation Buttons
Something I’d like LessWrong to do better is to allow authors to transition from hobbyists, to professionals that get paid to research and write full time.
Earlier this year, I was thinking about whether LessWrong should become more like substack, where there’s an easy affordance to start supporting financially supporting authors you like. I liked the idea but wasn’t sure it’d be healthy for LessWrong – the sorts of posts that make people excited to donate are often more tribal/political. But this seemed less worrisome during The Review. It’s a time when people are thinking holistically about the LessWrong intellectual world, comparing many different posts against each other and reflecting on which ones were truly valuable.
So, after the Final Vote this year, all posts above some[1] threshold will get a donation button interface, which makes it easier people to just give the author money. I encourage everyone to donate in proportion to how much value you got from a post. If it slightly improved your life, maybe donate $20-$50 as a thank you. If you think a post was a crucial insight for helping the entire world, maybe donate as if it were an effective altruism target. (i.e. if you’re the sort of person who donates 10% of your income, consider if any LessWrong posts are competitive with the other causes you might give to). LessWrong posts are a public good, and I think at least some are worth supporting in this way.
Lightcone Infrastructure will be allocating our own prizes. We have not decided the total amount we’ll give, but it will most likely be substantially more than the $2000 we awarded last year.
[1] I’m not yet sure exactly what threshold to set. I’m expecting a lot of mediocre posts to get at least one positive vote, which shouldn’t automatically warrant inclusion in donations list.
The 2019 Books
The 2018 books were well received last year, selling out almost the entire 4000 sets we printed (though there are still some 300+ copies in Australia, available on Amazon there).
The 2019 year’s books are a week or two away from launch. They include 59 essays, each of which has a unique customized illustration generated by machine learning. They’ll be eligible again for Amazon Prime, so shipping will be fast in North America, and likely in time for Christmas. A little later we’ll be supplying books to Amazon UK, which is where European readers can order from (with slightly longer shipping times and prices).
We will not ourselves be shipping to every other country – last year we attempted to ship to ~25 countries, most of which sold very few copies while requiring a lot of setup work. Alas, we are LessWrong, not Santa Claus – we unfortunately exist and are subject to logistical constraints. :P
At this time we’re not committed to doing another anthology set next year. We’re going to wait until after the launch of this year’s books to see whether there’s demand for annual anthologies. We have some different book projects in mind for the community, including a book of The Core Sequences, or entire sequences by other authors that fare well in the review, or books dedicated to a single topic drawing from the full history of LessWrong (covering topics such as Coordination or AI Alignment).
Meanwhile, we’ll definitely be collating the winning 2020 essays into a proper LessWrong sequence, prominently displayed in the site library. (I expect to have the 2018 and 2019 sequences released later this week). And again, we’ll be awarding significantly more financial prizes this year, and facilitating donation buttons to make it easier to reward authors who have done good work.
Here’s a sneak peak of the spines of the upcoming books, which includes this year’s volume titles (Book 1 is on the bottom). This year’s books are notably bigger than last year’s, 60% bigger in terms of page size.
Voting on Important Intellectual Progress
In past years, the vote was officially for creating a published book. This made it easier to reason about what exactly you were voting for, but also meant that some types of posts were harder conceptually to reward. Some important progress isn’t very fun to read. Some important posts are massively long, and couldn’t possibly fit in a book.
So this year, I’d like to formally ask that you vote based on how important an intellectual contribution a post made, rather than whether you think it makes sense to publish.
The LessWrong moderation team will take stock of the top-rated posts, and make judgment calls on how to best reward them. Some may fit best into anthology style books. Some may be more appropriate for (eventual) textbooks. Some might be important-but-tedious empirical work that makes more sense to give an honorable mention to in the books, while primarily rewarding them with prize money.
In practice, this is not that different from how we’ve been assembling the books in previous years. But it had been a bit ambiguous, and I thought it best to make it official.
You are welcome to use your own taste in what you consider important intellectual progress. But some questions that might inform your vote include:
Does this post introduce a concept that helps you understand the world?
Does the post provide useful and accurate empirical data?
Does this post teach a skill that has helped you?
Does this post summarize or distill information that makes it easier to grasp?
Do the central arguments of the post make sense?
Does this post promote an important and interesting hypothesis?
While writing reviews, it’s also worth exploring questions like:
How does this fit into the broader intellectual landscape?
What further work would you like to see?
Go Forth and Review!
I have more ideas for how to improve the Review this year, which I’ll be posting about as they reach fruition. Meanwhile, let the LessWrong 2020 Review commence!
Head over to your past upvotes page and start voting!
- The LessWrong 2022 Review by (5 Dec 2023 4:00 UTC; 115 points)
- Voting Results for the 2020 Review by (2 Feb 2022 18:37 UTC; 108 points)
- The LessWrong 2021 Review: Intellectual Circle Expansion by (1 Dec 2022 21:17 UTC; 95 points)
- Prizes for the 2020 Review by (20 Feb 2022 21:07 UTC; 94 points)
- Implications of Civilizational Inadequacy (reviewing mazes/simulacra/etc) by (21 Jan 2022 1:45 UTC; 93 points)
- 2020 Review Article by (14 Jan 2022 4:58 UTC; 74 points)
- 2020 Review: The Discussion Phase by (15 Dec 2021 1:12 UTC; 55 points)
- 2020 Review: Final Voting by (28 Jan 2022 3:39 UTC; 34 points)
- 's comment on 5 Tips for Good Hearting by (1 Apr 2022 19:48 UTC; 2 points)
- 's comment on Effective Altruism: The First Decade (Forum Review) by (EA Forum; 2 Dec 2021 1:00 UTC; 1 point)
Thank you for the link to my 2020 upvotes. I didn’t know that was a thing. It brings the preliminary voting up from “super inconvenient” to “convenient”.
It’s weird looking at my list of strong upvotes, given that a lot of them are posts I have no memory of. “I guess I really liked this post since I strong-upvoted it, also I guess it was forgettable since if you’d told me I’d never seen it, I might have believed you.”
(Possibly this says more about my memory than about the posts.)
Same. I think it comes at least partially from the fact that I can only either cast weak vote or a strong vote, and then even if I feel a post doesn’t deserve a full upvote, but it deserves it more than it deserves just a weak vote, then I’ll give it a strong upvote but somewhat reluctantly.
Also it can just be simple bad prediction. The post seems good and valuable when I first read it, but I end not thinking about it or using anything from it so I forget.
It is a thing as of today. :)
While going thru my Anki deck of insights and found some relevant LessWrong pages and upvoted those. While doing so I came across this quote I somehow discovered on 2020 in an old LessWrong quotes thread:
And while pondering it I thought: Wouldn’t it be nice to have some rationality quotes sprinkled across the books? There was no quotes thread in 2020 but such quotes can be timeless, right?
Rationality quotes would work best as chapter-start epigrams, I think.
I just made a significant update to the Review Dashboard page.
Most Important Structural Change:
Posts with at least one review, that you. haven’t voted on, get sorted to the top. This means that if you think a post is particularly worth people’s attention to vote on, you can get it in front of more eyeballs easily by writing at least a short review of it.
Other Changes:
The default view for the page includes recent Reviews, so it’s easier to catch up on new reviews
The Post List on the dashboard now shows posts with unread comments, so you can more easily click to read discussion.
The expanded-post-view on the Dashboard page now includes Pingbacks for each post, so you can see how it was followed up on over time.
Streamlined the explanation text so it was easier to read the important bits. You can mouseover individual questions in an FAQ to get more detail.
I got rid of nominations this year because they felt basically the same as positive reviews, and the last couple years it often felt redundant for two people to write nominations basically saying “this post was good for the obvious reasons why it’d be good”, and then to require someone else to later write a review saying “yup, this is still good for the obvious reasons why it’s good.”
But, one person who looked over the new UI said that they felt it was a little intimidating to write reviews compared to nominations, and there was something nice about authors getting comments from people-in-the-future who liked their work.
I just went and wrote some short nomination-style reviews to get the ball rolling. But I’m curious if anyone else feels like it’s a little harder to write them this year. I might go tweaking the UI a bit to try to make that feel like more of an accessible choice.
Reviewing this year was more fun and much easier than in previous years. Having a page of all the posts I’d voted on, plus the voting page showing me the posts others have reviewed, made it more like an inbox to me, and something I felt was easy to complete.
Problem for me is that, for whatever reason, I rarely cast karma votes and my bar in both directions is very high, so I didn’t have a reference at all.
I expect the quadratic voting not to be very different from the 1-4-9 system, but I favor including quadratic voting again even if that is the case. I have two actual reasons for this:
It’s a cool mechanism, with flexible levels of engagement, and this is a good way to practice using it. If we don’t make options like this available when voting opportunities arise, we can’t expect them to ever appear in critical arenas like elections or governance.
The more posts there are, the more valuable being able to fine-tune our votes becomes, operating under the assumption that the number of quality posts correlates with the number of posts overall (which I strongly expect). Since there are more posts this year, more granular voting has more value than it did last year. I want to be able to capture the additional value of the opportunity for granular voting.
I do think, on the margin, that experimenting with new mechanism design is pretty important. Whether to stick with a given mechanism depends a lot on whether that’s a mechanism I expect to really scale up.
The Lightcone team has used quadratic voting internally on occasion (i.e. for deciding which topics to discuss during a team retreat). But I’m not sure this was all that much better than a really simple averaging. I don’t know which domains really benefit from quadratic voting.
For comparison – I’ve also considered building something like Liquid Democracy into the LW Review, where you can delegate your vote to someone else.
Counterpoint: the quadratic voting mechanism was difficult to understand, and confusing. 1-4-9 is simple.
I’m legitimately unsure about the differences in complexity, actually. Quadratic Voting throws a bunch of upfront complexity at you, and forces you to deal with it. The impemntation of1-4-9 is secretly somewhat complex in that the exact value of your votes varies based on how many votes you cast of varying strengths.
I expect sometime-this-month to build UI that shows you what the actual value of your votes on hover-over, so you have a bit more visibility into what’s going on, but it’d still be a bit opaque and probably require more overall effort to “truly” understand than quadratic voting.
“Dah, I can’t vote for my own posts.”
looks at own posts from 2020.
“Well, 2019 was a better year anyhow.”
No sarcasm, though, I think I had only 2 good posts in 2020 (Gricean communication and meta-preferences, and Hierarchical planning: context agents), both received modest upvotes and little discussion, and will probably stay out of the 2020 review.
I am not sure that increasing the amount of engaging writing by people with good network effects would lead to more people digging around to comment on my 2 good posts per year, and so I am ambivalent about “going in the Substack direction” even if it works. But prizes for the best posts of 2020 sound neat.
For what it’s worth, I went through the 2020 posts, and found zero that I felt were worthy of noting or preservation.
What do the negative votes mean, exactly?
Negative votes are for conveying you think a post is some combination of misleading, harmful or unimportant.
(Also, FYI each vote has a hoverover tooltip)
Thanks! (Somehow my brain just glossed over the tooltips, even though I had to have seen them spawn to click on the votes…)
(… also, why the heck would anyone downvote my comment?!)
Damn, I thought the books were supposed to go up today and was looking all over for them. Glad I finally figured it out though!
Note: I’ve extended the nominations deadline by 1 day, because… I sadly had forgotten it was listed in UTC time, and it was coming right after Bay Solstice Celebration weekend and I’m a bit behind on getting everything set up for the Review Phase.
“The 2018 books were well received last year, selling out almost the entire 4000 sets we printed (though there are still some 300+ copies in Australia, available on Amazon there)”—I’m curious why you ordered so many for Australia as that seems much larger than the community here. Are you being charged for the continued storage of these books?
The reason why we ordered so many books there is because we didn’t really think about how small Australia is, and had to make the decision on a somewhat short deadline with a bunch of other stuff happening at the same time. I think if we had thought about it for longer, we wouldn’t have made that mistake.
The ‘Less Wrong Melbourne’ facebook group has 563 members—I have no idea how many are currently active, nor what proportion of the LW readership joins facebook groups (I for one am not a member of the group or even a facebook user, I just googled it out of curiosity), but it seems like a decent number, perhaps enough to make selling a few hundred books to Australians a reasonable expectation.
Facebook group membership is an extremely weak commitment
Maybe joining the Sydney Meetup group is a stronger signal? That’s at 459 members, and presumably those are people who care enough to seriously consider attending in-person events. Which would be a pretty small fraction of total readers located in the area.
The books section could be clarified, as to whether it’s {still | no longer} possible to order books from 2019. I’d love to order some.
The 2019 books haven’t been released yet (it’ll be possible to order them starting in a week or two). The 2018 books should be available if you want to order them in Australia.
I’m cautiously optimistic about the possibility of LessWrong enabling people to write professionally.
I’m weakly skeptical that Substack-style crowdfunding is the way to go.
I won’t be donating; I use my marginal dollars more altruistically. I think it’s fine for people to donate, of course, but such donations generally don’t make sense to come out of your effective altruism budget, since their counterfactual impact is quite small. If you think incentivizing excellent LessWrong posts is highly effective, then it would be better to publicly promise to donate [that much] to the authors of your favorite posts in 2022. Or if you’re too modest for that, PM me and I’ll publicly announce that an anonymous person has promised to donate [that much] next year. (Of course, donating this year to thank the author is fine unless you’re a fanatical effective altruist — but it’s prima facie not highly effective.)
Why is that the case? Is it just that people can’t see how much you’ve donated via donation buttons? I assume that some aggregate donation figures will be made public later on, though, so making those figures higher seems pretty similar to you announcing donations personally.
Fair, I was implicitly assuming that no donation figures would be public.
I’m unsure exactly how much we’ll be making public, but I do expect at least aggregate donations to be public. The entire point here is to not merely have a few people committing to give money, but to build an entire system that helps authors have a justified expectation that important posts generally get money. And among the more credible ways to signal this is going to happen in the future is to start doing it now.
(I expect to have figured out how public things will be by the time we get to the Final Voting phase)
I am very much in favor of turning the core sequences into a series of books!
Is there still any chance of getting my hands on the 2018 books? I live in Europe so ordering from Australian Amazon is not an option.
On a minor point, thank you for making the 2019 books have bigger page size! While there’s something neat about having little pocket-sized books (cf OUP’s Very Short Introductions), and I have good eyesight, some people would have found the print really hard to read.
Curious if you know anyone who specifically ran into the ‘hard to read’ problem? (I find it hard to read, but had been surprised we hadn’t gotten more explicit complaints)
No (though I don’t personally know anyone else who has the books).
It’s of course a matter of degree—I can read them, but small print makes them less easy for me to read. Though I also wonder if it may subtly affect people’s comprehension, without them realizing it. As is the case when sight-reading sheet music that’s printed small or in poor light: you make more mistakes, without necessarily realizing why.