A worry I have with our voting system this year is that it felt more like “ranking posts by prestige” than “ranking them by truth/usefulness.” It so happens that “prestige in LW community” does prioritize truth/usefulness and I think the outcome mostly tracked that, but I think we can do better.
The reason I’m worried is a) I’d expected, by default, for these things to be jumbled together, and b) whatever you thought of Affordance Widths, it seems pretty unlikely for it’s “-140” score to be based on the merits of the post, rather than people not wanting the author represented in a Best of LW book.
I think reputational effects matter and it’s fine to give people an outlet for that, but I think it’s better if we ask those questions separately from questions of truth, and usefulness. (And, I think the vote and overall Review process is more interesting if truth/usefulness is the primary thing getting looked into)
It so happens I don’t think Affordance Widths was in the top 35 posts, and not something I’d have included in the book on its merits (not because it was wrong – it seems like a basically true model to me – just because there was other stuff that was better). I think it’s plausible for the post to have ended up with a negative score on it’s own merits, but I doubt that an author-blind review would have given it −140.
So right now, I’d assume that we’d get similar results from any post by a similarly unpopular author, even if the ideas in the post were quite important.
I think it’s pretty important for the community to be able to engage with that sort of thing, and maybe collectively decide “okay, we’re not putting this in our public facing book for reputational reasons, but we should at least be able to evaluate it clearly in our inward facing review process.”
And meanwhile, the fact that that post’s score was clearly determined reputationally reinforces some of my prior-worries about the vote getting truth and prestige jumbled together, in less controversial post examples.
I had been worried about this earlier in the design-process for the review. A major reason I didn’t object to the system we ended up with is...
...well, if you tell people “evaluate these posts for truth/usefulness”, but clearly the results of the vote are going to translate into prestige, it seems like it makes the situation worse rather than better if you try to pretend otherwise.
But I think I might be interested in experimenting in another direction next year, where the vote isn’t focused on relative ranking of posts at all, instead it’s more like a survey where people answer qualitative questions, like:
Have you thought about the ideas in this post in the past year?
Do the ideas in this post seem important?
How do you feel about this post’s epistemics
Should this post appear in a public-facing Best of LW book?
Should this post appear in an inward-facing, high-context LW Journal?
Truth/Usefulness vs Prestige/Reputation
A worry I have with our voting system this year is that it felt more like “ranking posts by prestige” than “ranking them by truth/usefulness.” It so happens that “prestige in LW community” does prioritize truth/usefulness and I think the outcome mostly tracked that, but I think we can do better.
The reason I’m worried is a) I’d expected, by default, for these things to be jumbled together, and b) whatever you thought of Affordance Widths, it seems pretty unlikely for it’s “-140” score to be based on the merits of the post, rather than people not wanting the author represented in a Best of LW book.
I think reputational effects matter and it’s fine to give people an outlet for that, but I think it’s better if we ask those questions separately from questions of truth, and usefulness. (And, I think the vote and overall Review process is more interesting if truth/usefulness is the primary thing getting looked into)
It so happens I don’t think Affordance Widths was in the top 35 posts, and not something I’d have included in the book on its merits (not because it was wrong – it seems like a basically true model to me – just because there was other stuff that was better). I think it’s plausible for the post to have ended up with a negative score on it’s own merits, but I doubt that an author-blind review would have given it −140.
So right now, I’d assume that we’d get similar results from any post by a similarly unpopular author, even if the ideas in the post were quite important.
I think it’s pretty important for the community to be able to engage with that sort of thing, and maybe collectively decide “okay, we’re not putting this in our public facing book for reputational reasons, but we should at least be able to evaluate it clearly in our inward facing review process.”
And meanwhile, the fact that that post’s score was clearly determined reputationally reinforces some of my prior-worries about the vote getting truth and prestige jumbled together, in less controversial post examples.
I had been worried about this earlier in the design-process for the review. A major reason I didn’t object to the system we ended up with is...
...well, if you tell people “evaluate these posts for truth/usefulness”, but clearly the results of the vote are going to translate into prestige, it seems like it makes the situation worse rather than better if you try to pretend otherwise.
But I think I might be interested in experimenting in another direction next year, where the vote isn’t focused on relative ranking of posts at all, instead it’s more like a survey where people answer qualitative questions, like:
Have you thought about the ideas in this post in the past year?
Do the ideas in this post seem important?
How do you feel about this post’s epistemics
Should this post appear in a public-facing Best of LW book?
Should this post appear in an inward-facing, high-context LW Journal?