Even though you say some things in the direction already, I want to harp on the distinction between what we can judge by looking at a post in itself and things which can and should only be judged by the test of time.
I am thinking of a particular essay (not on LW) about how peer review should not judge the significance of a work, only the accuracy, but I can’t find it. I think “something like that” is central to your point, since you
want something like a retrospective judgement about which essays have stood the test of time
also want to have more features to lower the bar.
The essay I am thinking of was about a publishing venue which was established with the explicit goal of peer reviewers judging the rigor of a paper but not its impact/significance, since this cannot be judged ahead of time and is not very relevant to whether work should be published (sort of a generalization of the way replications are harder to publish—facts are facts, and should be publishable regardless of how flashy they are). The scientist was complaining that a paper of theirs was rejected from that venue due to not being of sufficient interest to the readership. The question of impact had been allowed to creep back into the reviews.
I think there’s a very general phenomenon where a venue starts out “alive”—able to spontaneously generate ideas—and then produces some good stuff, which raises the expectations, killing off the spontaneity. This can happen with individuals or groups. Some people start new twitter accounts all the time because there is too much pressure to perform once an account has gotten many followers. Old LW split into Discussion and Main, and then Discussion split off discussion threads to indicate a lowered bar.
Um. I guess I’m not being very solution-oriented here.
The problem is, something seems to implicitly raise people’s standards as a place gets good, even if there are explicit statements to the contrary (like the rule stating peer reviewers should not judge impact). This can kill a place over time.
Carefully separating what can be judged in the moment vs only after the fact seems like a part of the solution.
Maybe you want to create a “comfortable sloshing mess” of relatively low signal-to-noise chatter which makes people feel comfortable to post, while also having the carefully determined canon which contains the value.
Obviously the “comfortable sloshing mess” is not distinguished primarily by its being bad—it should be good, but good along different dimensions than the canon. It should meet high standards of rigor along dimensions that are easy to judge from the text itself and not difficult or off-putting for writers to meet. There should be a “google docs comment peer review” for these aspects. (Maybe not **only** for these aspects, but somehow virtuously aligned with respect to these aspects?)
Even though you say some things in the direction already, I want to harp on the distinction between what we can judge by looking at a post in itself and things which can and should only be judged by the test of time.
I am thinking of a particular essay (not on LW) about how peer review should not judge the significance of a work, only the accuracy, but I can’t find it. I think “something like that” is central to your point, since you
want something like a retrospective judgement about which essays have stood the test of time
also want to have more features to lower the bar.
The essay I am thinking of was about a publishing venue which was established with the explicit goal of peer reviewers judging the rigor of a paper but not its impact/significance, since this cannot be judged ahead of time and is not very relevant to whether work should be published (sort of a generalization of the way replications are harder to publish—facts are facts, and should be publishable regardless of how flashy they are). The scientist was complaining that a paper of theirs was rejected from that venue due to not being of sufficient interest to the readership. The question of impact had been allowed to creep back into the reviews.
I think there’s a very general phenomenon where a venue starts out “alive”—able to spontaneously generate ideas—and then produces some good stuff, which raises the expectations, killing off the spontaneity. This can happen with individuals or groups. Some people start new twitter accounts all the time because there is too much pressure to perform once an account has gotten many followers. Old LW split into Discussion and Main, and then Discussion split off discussion threads to indicate a lowered bar.
Um. I guess I’m not being very solution-oriented here.
The problem is, something seems to implicitly raise people’s standards as a place gets good, even if there are explicit statements to the contrary (like the rule stating peer reviewers should not judge impact). This can kill a place over time.
Carefully separating what can be judged in the moment vs only after the fact seems like a part of the solution.
Maybe you want to create a “comfortable sloshing mess” of relatively low signal-to-noise chatter which makes people feel comfortable to post, while also having the carefully determined canon which contains the value.
Obviously the “comfortable sloshing mess” is not distinguished primarily by its being bad—it should be good, but good along different dimensions than the canon. It should meet high standards of rigor along dimensions that are easy to judge from the text itself and not difficult or off-putting for writers to meet. There should be a “google docs comment peer review” for these aspects. (Maybe not **only** for these aspects, but somehow virtuously aligned with respect to these aspects?)