Currently, a post can appear in this section if … it has a score of at least 50 …
Is this adjusted by post date? Posts from before the relaunch are going to have much less karma, on average (and as user karma grows and the karma weight of upvotes grows with it, average karma will increase further). A post from last month with 50 karma, and a post from 2010 with 50 karma, are really not comparable…
We manually exclude posts if they aged poorly in a way that wouldn’t be captured by votes at the time—for example … reporting of studies that later failed to replicate
I wish you wouldn’t!
It seems to me that it would be extremely valuable to include posts like this in the recommendations—but annotate them with a note that the research in question hasn’t replicated. This would, I think, have an excellent pedagogic effect! To see how popular, how highly-upvoted, a study could be, while turning out later to have been bunk—think of the usefulness as a series of naturalistic rationality case studies! (Likewise useful would be to examine the comment threads of these old posts; did any of the commentariat suspect anything amiss? If so, what heuristics did they use? Did certain people consistently get it right, and if so, how? etc.) The new recommendation engine could do great good, in this way…
Is this adjusted by post date? Posts from before the relaunch are going to have much less karma, on average (and as user karma grows and the karma weight of upvotes grows with it, average karma will increase further). A post from last month with 50 karma, and a post from 2010 with 50 karma, are really not comparable…
This is one of a number of significant problems with using karma for this. My ideal system—which we probably won’t do soon, because of the amount of effort involved—would be something like:
Periodically, users get a list of posts that they read over the past week, end are asked to pick their favorite and to update their votes
This is converted into pairwise comparisons end used to generate an elo rating for each post
The recommender has a VOI factor to increase the visibility of posts where it doesn’t have a precise enough estimate of the rating
We separately have trusted raters compare posts from a more random sampling, compute a separate set of ratingr that way, and use it as a ground truth to set the tuning parameters and see how well it’s working.
In this world, karma would still be displayed and updated in response to votes the same way it is now, to give people an estimate of visibility and reception and to get a quick initial estimate of quality, but it would be superseded as a measurement of post quality for older content.
It seems to me that it would be extremely valuable to include posts like this in the recommendations—but annotate them with a note that the research in question hasn’t replicated. This would, I think, have an excellent pedagogic effect! To see how popular, how highly-upvoted, a study could be, while turning out later to have been bunk—think of the usefulness as a series of naturalistic rationality case studies! (Likewise useful would be to examine the comment threads of these old posts; did any of the commentariat suspect anything amiss? If so, what heuristics did they use? Did certain people consistently get it right, and if so, how? etc.) The new recommendation engine could do great good, in this way…
This is an interesting point. I think I would be in favor of this if we had a way to pin comments to the top as moderators. Right now I expect we could leave a comment, but I don’t expect that comment to actually show up high enough in the comment tree to be seen by most users, and we could edit the post but I am particularly hesitant to write retraction notices for other people.
Ideally I would want a way for things like this to happen organically driven by user activity instead of moderator intervention, but I don’t know yet how to best do that. Interested in suggestions, since it feels important for the broader vision of making progress over a long period of time.
I think there is still a loss of ownership that people would feel when we add big moderator note’s to the top of their posts, even if clearly signaled as moderator-added content, that I think would feel quite violating to many authors, though I might be wrong here.
I confess I don’t really know what you mean by this.
Not sure how to explain more. It would be good if there was some system that would allow other users that are not moderators to be able to inform other users about the updated epistemic content of a post. There are many potential ways to achieve that.
One might be to add inline comments that when they reach a certain threshold of votes can be displayed prominently enough to get the attention of others reading the content for the first time (though that also comes with cost), another might be to find some way to reduce or remove the strong first-mover bias in comment sections that prevent new comments from reaching the top of the comment section most of the time (due to voting activity usually being concentrated right after a post is created, which makes it hard fo rnew comments to get a lot of upvotes).
It would be good if there was some system that would allow other users that are not moderators to be able to inform other users about the updated epistemic content of a post.
I see, yes. Well, I agree that such a system would be good to have, but I am not convinced that it would be better for what I have in mind that using the recommendation system you’ve built for this. After all, three-quarters of the work here is precisely in bringing the old posts in question to the attention of users; relying on users in the first place, to accomplish that, seems to be an ineffective plan—whereas using the automated recommendation engine is perfect. (Still the user-originated system you allude to would, I think, be a good supplement.)
I think there is still a loss of ownership that people would feel when we add big moderator note’s to the top of their posts, even if clearly signaled as moderator-added content, that I think would feel quite violating to many authors, though I might be wrong here.
Well, that seems to me to be a matter of designing the UI/styling for clear separation, which is an eminently tractable problem. (Or do you disagree, with either clause?) There is, after all, all sorts of metadata and navigation UI and so on around a post, which is not generated by the author (directly or at all); have the layout and styling and such of these “moderator notes” clearly associate them with this metadata/navigation, and I think (unless I am misunderstanding you) that your concern is thereby addressed.
After all, three-quarters of the work here is precisely in bringing the old posts in question to the attention of users; relying on users in the first place, to accomplish that, seems to be an ineffective plan—whereas using the automated recommendation engine is perfect. (Still the user-originated system you allude to would, I think, be a good supplement.)
This indicates at least some misunderstanding of what I tried to convey. I agree that the recommendation system can do the job of promoting the visibility of such posts, but then I was additionally suggesting that it would be good to independently allow users to promote epistemic corrections to a higher level of visibility on the post-page itself in a way that does not require moderator interaction.
I think agree that we can do some better UI work to show that separation, and I think that’s probably the correct long-term strategy. Just the backlog of additional features like that is long, and difficulty of solving this problem well isn’t trivial (and neither is the cost of messing up), so I was mostly comparing options that don’t require any additional features like that and keep the existing site hierarchy.
This discussion has however made me update that putting in the relevant effort does surface a good amount of additional value, so I will think about that more.
Is this adjusted by post date? Posts from before the relaunch are going to have much less karma, on average (and as user karma grows and the karma weight of upvotes grows with it, average karma will increase further). A post from last month with 50 karma, and a post from 2010 with 50 karma, are really not comparable…
Rerunning the whole vote history with the new karma is one of the next things on our to-do list. Right now it will indeed be biased towards the recent year, which I hope to fix soon (that is one of the things that I consider necessary before removing the “[beta]” tag from the feature).
Is this adjusted by post date? Posts from before the relaunch are going to have much less karma, on average (and as user karma grows and the karma weight of upvotes grows with it, average karma will increase further). A post from last month with 50 karma, and a post from 2010 with 50 karma, are really not comparable…
I wish you wouldn’t!
It seems to me that it would be extremely valuable to include posts like this in the recommendations—but annotate them with a note that the research in question hasn’t replicated. This would, I think, have an excellent pedagogic effect! To see how popular, how highly-upvoted, a study could be, while turning out later to have been bunk—think of the usefulness as a series of naturalistic rationality case studies! (Likewise useful would be to examine the comment threads of these old posts; did any of the commentariat suspect anything amiss? If so, what heuristics did they use? Did certain people consistently get it right, and if so, how? etc.) The new recommendation engine could do great good, in this way…
This is one of a number of significant problems with using karma for this. My ideal system—which we probably won’t do soon, because of the amount of effort involved—would be something like:
Periodically, users get a list of posts that they read over the past week, end are asked to pick their favorite and to update their votes
This is converted into pairwise comparisons end used to generate an elo rating for each post
The recommender has a VOI factor to increase the visibility of posts where it doesn’t have a precise enough estimate of the rating
We separately have trusted raters compare posts from a more random sampling, compute a separate set of ratingr that way, and use it as a ground truth to set the tuning parameters and see how well it’s working.
In this world, karma would still be displayed and updated in response to votes the same way it is now, to give people an estimate of visibility and reception and to get a quick initial estimate of quality, but it would be superseded as a measurement of post quality for older content.
This is an interesting point. I think I would be in favor of this if we had a way to pin comments to the top as moderators. Right now I expect we could leave a comment, but I don’t expect that comment to actually show up high enough in the comment tree to be seen by most users, and we could edit the post but I am particularly hesitant to write retraction notices for other people.
Ideally I would want a way for things like this to happen organically driven by user activity instead of moderator intervention, but I don’t know yet how to best do that. Interested in suggestions, since it feels important for the broader vision of making progress over a long period of time.
Why not insert a note at the top of the post?
Make it stand out, visually, like put it in a “moderator note” box or whatever, and you’re good to go…
I confess I don’t really know what you mean by this.
I think there is still a loss of ownership that people would feel when we add big moderator note’s to the top of their posts, even if clearly signaled as moderator-added content, that I think would feel quite violating to many authors, though I might be wrong here.
Not sure how to explain more. It would be good if there was some system that would allow other users that are not moderators to be able to inform other users about the updated epistemic content of a post. There are many potential ways to achieve that.
One might be to add inline comments that when they reach a certain threshold of votes can be displayed prominently enough to get the attention of others reading the content for the first time (though that also comes with cost), another might be to find some way to reduce or remove the strong first-mover bias in comment sections that prevent new comments from reaching the top of the comment section most of the time (due to voting activity usually being concentrated right after a post is created, which makes it hard fo rnew comments to get a lot of upvotes).
I see, yes. Well, I agree that such a system would be good to have, but I am not convinced that it would be better for what I have in mind that using the recommendation system you’ve built for this. After all, three-quarters of the work here is precisely in bringing the old posts in question to the attention of users; relying on users in the first place, to accomplish that, seems to be an ineffective plan—whereas using the automated recommendation engine is perfect. (Still the user-originated system you allude to would, I think, be a good supplement.)
Well, that seems to me to be a matter of designing the UI/styling for clear separation, which is an eminently tractable problem. (Or do you disagree, with either clause?) There is, after all, all sorts of metadata and navigation UI and so on around a post, which is not generated by the author (directly or at all); have the layout and styling and such of these “moderator notes” clearly associate them with this metadata/navigation, and I think (unless I am misunderstanding you) that your concern is thereby addressed.
This indicates at least some misunderstanding of what I tried to convey. I agree that the recommendation system can do the job of promoting the visibility of such posts, but then I was additionally suggesting that it would be good to independently allow users to promote epistemic corrections to a higher level of visibility on the post-page itself in a way that does not require moderator interaction.
Ah! Yes, I understand now, and entirely agree!
I think agree that we can do some better UI work to show that separation, and I think that’s probably the correct long-term strategy. Just the backlog of additional features like that is long, and difficulty of solving this problem well isn’t trivial (and neither is the cost of messing up), so I was mostly comparing options that don’t require any additional features like that and keep the existing site hierarchy.
This discussion has however made me update that putting in the relevant effort does surface a good amount of additional value, so I will think about that more.
Rerunning the whole vote history with the new karma is one of the next things on our to-do list. Right now it will indeed be biased towards the recent year, which I hope to fix soon (that is one of the things that I consider necessary before removing the “[beta]” tag from the feature).