Is this adjusted by post date? Posts from before the relaunch are going to have much less karma, on average (and as user karma grows and the karma weight of upvotes grows with it, average karma will increase further). A post from last month with 50 karma, and a post from 2010 with 50 karma, are really not comparable…
This is one of a number of significant problems with using karma for this. My ideal system—which we probably won’t do soon, because of the amount of effort involved—would be something like:
Periodically, users get a list of posts that they read over the past week, end are asked to pick their favorite and to update their votes
This is converted into pairwise comparisons end used to generate an elo rating for each post
The recommender has a VOI factor to increase the visibility of posts where it doesn’t have a precise enough estimate of the rating
We separately have trusted raters compare posts from a more random sampling, compute a separate set of ratingr that way, and use it as a ground truth to set the tuning parameters and see how well it’s working.
In this world, karma would still be displayed and updated in response to votes the same way it is now, to give people an estimate of visibility and reception and to get a quick initial estimate of quality, but it would be superseded as a measurement of post quality for older content.
This is one of a number of significant problems with using karma for this. My ideal system—which we probably won’t do soon, because of the amount of effort involved—would be something like:
Periodically, users get a list of posts that they read over the past week, end are asked to pick their favorite and to update their votes
This is converted into pairwise comparisons end used to generate an elo rating for each post
The recommender has a VOI factor to increase the visibility of posts where it doesn’t have a precise enough estimate of the rating
We separately have trusted raters compare posts from a more random sampling, compute a separate set of ratingr that way, and use it as a ground truth to set the tuning parameters and see how well it’s working.
In this world, karma would still be displayed and updated in response to votes the same way it is now, to give people an estimate of visibility and reception and to get a quick initial estimate of quality, but it would be superseded as a measurement of post quality for older content.