Why are so many resources being sunk into this specifically? I just don’t understand how it makes sense, what the motivation is and how they arrived at the idea. Maybe there is a great explanation and thought process which I am missing.
From my point of view, there is little demand for it and the main motivation might plausibly have been “we want to say we’ve published a book” rather than something that people want or need.
Having said that, I’d rather get an answer to my initial comment—why it makes sense to you/them—rather than me having to give reasons why I don’t see how it makes sense.
We have written some things about our motivation on this, though I don’t think we’ve been fully comprehensive by any means (since that itself would have increased the cost of the vote a good amount). Here are the posts that we’ve written on the review and the motivation behind it:
We need to build on each other’s intellectual contributions, archive important content, and avoid primarily being news-driven.
We need to improve the signal-to-noise ratio for the average reader, and only broadcast the most important writing
[...]
Modern science is plagued by severe problems, but of humanity’s institutions it has perhaps the strongest record of being able to build successfully on its previous ideas.
The physics community has this system where the new ideas get put into journals, and then eventually if they’re important, and true, they get turned into textbooks, which are then read by the upcoming generation of physicists, who then write new papers based on the findings in the textbooks. All good scientific fields have good textbooks, and your undergrad years are largely spent reading them.
Over the past couple years, much of my focus has been on the early-stages of LessWrong’s idea pipeline – creating affordance for off-the-cuff conversation, brainstorming, and exploration of paradigms that are still under development (with features like shortform and moderation tools).
But, the beginning of the idea-pipeline is, well, not the end.
I’ve written a couple times about what the later stages of the idea-pipeline might look like. My best guess is still something like this:
I want LessWrong to encourage extremely high quality intellectual labor. I think the best way to go about this is through escalating positive rewards, rather than strong initial filters.
Right now our highest reward is getting into the curated section, which… just isn’t actually that high a bar. We only curate posts if we think they are making a good point. But if we set the curated bar at “extremely well written and extremely epistemically rigorous and extremely useful”, we would basically never be able to curate anything.
My current guess is that there should be a “higher than curated” level, and that the general expectation should be that posts should only be put in that section after getting reviewed, scrutinized, and most likely rewritten at least once.
I still have a lot of uncertainty about the right way to go about a review process, and various members of the LW team have somewhat different takes on it.
I’ve heard lots of complaints about mainstream science peer review: that reviewing is often a thankless task; the quality of review varies dramatically, and is often entangled with weird political games.
------
Before delving into the process, I wanted to go over the high level goals for the project:
1. Improve our longterm incentives, feedback, and rewards for authors
2. Create a highly curated “Best of 2018” sequence / physical book
3. Create common knowledge about the LW community’s collective epistemic state regarding controversial posts
-------
Longterm incentives, feedback and rewards
Right now, authors on LessWrong are rewarded essentially by comments, voting, and other people citing their work. This is fine, as things go, but has a few issues:
Some kinds of posts are quite valuable, but don’t get many comments (and these disproportionately tend to be posts that are more proactively rigorous, because there’s less to critique, or critiquing requires more effort, or building off the ideas requires more domain expertise)
By contrast, comments and voting both nudge people towards posts that are clickbaity and controversial.
Once posts have slipped off the frontpage, they often fade from consciousness. I’m excited for a LessWrong that rewards Long Content, that stand the tests of time, as is updated as new information comes to light. (In some cases this may involve editing the original post. But if you prefer old posts to serve as a time-capsule of your post beliefs, adding a link to a newer post would also work)
Many good posts begin with an “epistemic status: thinking out loud”, because, at the time, they were just thinking out loud. Nonetheless, they turn out to be quite good. Early-stage brainstorming is good, but if 2 years later the early-stage-brainstorming has become the best reference on a subject, authors should be encouraged to change that epistemic status and clean up the post for the benefit of future readers.
The aim of the Review is to address those concerns by:
Promoting old, vetted content directly on the site.
Awarding prizes not only to authors, but to reviewers. It seems important to directly reward high-effort reviews that thoughtfully explore both how the post could be improved, and how it fits into the broader intellectual ecosystem. (At the same time, not having this be the final stage in the process, since building an intellectual edifice requires four layers of ongoing conversation)
Compiling the results into a physical book. I find there’s something… literally weighty about having your work in printed form. And because it’s much harder to edit books than blogposts, the printing gives authors an extra incentive to clean up their past work or improve the pedagogy.
------
Common knowledge about the LW community’s collective epistemic state regarding controversial posts
Some posts are highly upvoted because everyone agrees they’re true and important. Other posts are upvoted because they’re more like exciting hypotheses. There’s a lot of disagreement about which claims are actually true, but that disagreement is crudely measured in comments from a vocal minority.
The end of the review process includes a straightforward vote on which posts seem (in retrospect), useful, and which seem “epistemically sound”. This is not the end of the conversation about which posts are making true claims that carve reality at it’s joints, but my hope is for it to ground that discussion in a clearer group-epistemic state.
Further Comments
I expect we will write some more in the future about some of the broader goals behind the review, but the above I think summarizes a bunch of the high-level considerations reasonably well.
I think one way one could describe at least my motivation for the review is that one of the big holes that I’ve always perceived in LessWrong, and the internet at large, is the focus on things that are popular in the moment, and that it’s hard for people to really build on other people’s ideas and make long-term intellectual progres. The review is an experiment in creating an incentive and attention allocation mechanism that tries to counteract those forces. I am not yet sure how much it succeeded at that, though I am broadly pleased with how it went.
Why are so many resources being sunk into this specifically? I just don’t understand how it makes sense, what the motivation is and how they arrived at the idea. Maybe there is a great explanation and thought process which I am missing.
From my point of view, there is little demand for it and the main motivation might plausibly have been “we want to say we’ve published a book” rather than something that people want or need.
Having said that, I’d rather get an answer to my initial comment—why it makes sense to you/them—rather than me having to give reasons why I don’t see how it makes sense.
We have written some things about our motivation on this, though I don’t think we’ve been fully comprehensive by any means (since that itself would have increased the cost of the vote a good amount). Here are the posts that we’ve written on the review and the motivation behind it:
The LessWrong 2018 Review
Voting Phase of 2018 LW Review
(Feedback Request) Quadratic voting for the 2018 Review
The Review Phase
The first post includes more of our big-picture motivation for this. Here are some of the key quotes:
Quotes
In his LW 2.0 Strategic Overview, habryka noted:
Over the past couple years, much of my focus has been on the early-stages of LessWrong’s idea pipeline – creating affordance for off-the-cuff conversation, brainstorming, and exploration of paradigms that are still under development (with features like shortform and moderation tools).
But, the beginning of the idea-pipeline is, well, not the end.
I’ve written a couple times about what the later stages of the idea-pipeline might look like. My best guess is still something like this:
I still have a lot of uncertainty about the right way to go about a review process, and various members of the LW team have somewhat different takes on it.
I’ve heard lots of complaints about mainstream science peer review: that reviewing is often a thankless task; the quality of review varies dramatically, and is often entangled with weird political games.
------
Before delving into the process, I wanted to go over the high level goals for the project:
1. Improve our longterm incentives, feedback, and rewards for authors
2. Create a highly curated “Best of 2018” sequence / physical book
3. Create common knowledge about the LW community’s collective epistemic state regarding controversial posts
-------
Longterm incentives, feedback and rewards
Right now, authors on LessWrong are rewarded essentially by comments, voting, and other people citing their work. This is fine, as things go, but has a few issues:
Some kinds of posts are quite valuable, but don’t get many comments (and these disproportionately tend to be posts that are more proactively rigorous, because there’s less to critique, or critiquing requires more effort, or building off the ideas requires more domain expertise)
By contrast, comments and voting both nudge people towards posts that are clickbaity and controversial.
Once posts have slipped off the frontpage, they often fade from consciousness. I’m excited for a LessWrong that rewards Long Content, that stand the tests of time, as is updated as new information comes to light. (In some cases this may involve editing the original post. But if you prefer old posts to serve as a time-capsule of your post beliefs, adding a link to a newer post would also work)
Many good posts begin with an “epistemic status: thinking out loud”, because, at the time, they were just thinking out loud. Nonetheless, they turn out to be quite good. Early-stage brainstorming is good, but if 2 years later the early-stage-brainstorming has become the best reference on a subject, authors should be encouraged to change that epistemic status and clean up the post for the benefit of future readers.
The aim of the Review is to address those concerns by:
Promoting old, vetted content directly on the site.
Awarding prizes not only to authors, but to reviewers. It seems important to directly reward high-effort reviews that thoughtfully explore both how the post could be improved, and how it fits into the broader intellectual ecosystem. (At the same time, not having this be the final stage in the process, since building an intellectual edifice requires four layers of ongoing conversation)
Compiling the results into a physical book. I find there’s something… literally weighty about having your work in printed form. And because it’s much harder to edit books than blogposts, the printing gives authors an extra incentive to clean up their past work or improve the pedagogy.
------
Common knowledge about the LW community’s collective epistemic state regarding controversial posts
Some posts are highly upvoted because everyone agrees they’re true and important. Other posts are upvoted because they’re more like exciting hypotheses. There’s a lot of disagreement about which claims are actually true, but that disagreement is crudely measured in comments from a vocal minority.
The end of the review process includes a straightforward vote on which posts seem (in retrospect), useful, and which seem “epistemically sound”. This is not the end of the conversation about which posts are making true claims that carve reality at it’s joints, but my hope is for it to ground that discussion in a clearer group-epistemic state.
Further Comments
I expect we will write some more in the future about some of the broader goals behind the review, but the above I think summarizes a bunch of the high-level considerations reasonably well.
I think one way one could describe at least my motivation for the review is that one of the big holes that I’ve always perceived in LessWrong, and the internet at large, is the focus on things that are popular in the moment, and that it’s hard for people to really build on other people’s ideas and make long-term intellectual progres. The review is an experiment in creating an incentive and attention allocation mechanism that tries to counteract those forces. I am not yet sure how much it succeeded at that, though I am broadly pleased with how it went.
They’ve sold 1000 copies. I’m surprised, too!
Sure. I explained my personal enthusiasm for the Review in a November comment.