We’ve had the choice of tabs up for a month now and the results so far are encouraging, or at least not discouraging. There are many users who are very pleased with the Recommendations, liking among other things that it brings to attention posts that otherwise get lost if you only see what’s new. Clickthrough-rates are higher for people using the Enriched/Recommendations tab, although this is most certainly a selection effect on the kind of user who changes tab at all. Switching some people over automatically is motivated by wanting to get a better signal here before doing something like changing the global default.
The current recommendations still needs more work though. People are much less likely to click on recommendations of posts that they’ve already clicked on, but it’s proving tricky to eliminate such recommendation entirely. Also the algorithm overwhelmingly recommends posts from the last year when we’d like to see it surfacing stuff from further back too. Still, Latest is overwhelming stuff from the Last week, so it’s still an improvement over the counterfactual.
--
From when we started the project, we’ve settled on the “hybrid” list being likely optimal as the default list people look at. Many people want to “keep up with the latest” even if they’re also interested in good posts from all time, so any recommended list of posts that’s the default has to have a heavy latest component. We first tried making two calls to the Recommendations API, one with heavy recency bias, but it was hard to get it consisted, so we switched to just splitting the list between the usual Latest algorithm and new recommendations algorithm.
This has the advantage that is preserves some of the “common knowledge” aspect of the current algorithm where you know which posts other people are seeing too, and an author knows that if they get upvoted, their post will be visible automatically and transparently to many people. As discussed elsethread on this post, we want to have a pure-recommendations tab as well and have been waiting on a bit of coding to make that happen.
--
People often have the fear of goodharting on the wrong metric (like clicks) for recommendation algorithms. I think we do need to keep an eye on that, and I want to build more analytics tools for detecting drift here, and more talking to people. I think as we fix up more basic issues like excluding read content and getting it to even recommend posts from older than a year ago[1], we’ll put more attention on is the trend good.
One guess I have is the algorithm is stuck for dumb “structural” reasons, in that it’s been given recent data which is overwhelmingly of people reading recent content, so when it queries “what’s good?” recent content comes out on top even without explicitly training that into the system.
We’ve had the choice of tabs up for a month now and the results so far are encouraging, or at least not discouraging. There are many users who are very pleased with the Recommendations, liking among other things that it brings to attention posts that otherwise get lost if you only see what’s new. Clickthrough-rates are higher for people using the Enriched/Recommendations tab, although this is most certainly a selection effect on the kind of user who changes tab at all. Switching some people over automatically is motivated by wanting to get a better signal here before doing something like changing the global default.
The current recommendations still needs more work though. People are much less likely to click on recommendations of posts that they’ve already clicked on, but it’s proving tricky to eliminate such recommendation entirely. Also the algorithm overwhelmingly recommends posts from the last year when we’d like to see it surfacing stuff from further back too. Still, Latest is overwhelming stuff from the Last week, so it’s still an improvement over the counterfactual.
--
From when we started the project, we’ve settled on the “hybrid” list being likely optimal as the default list people look at. Many people want to “keep up with the latest” even if they’re also interested in good posts from all time, so any recommended list of posts that’s the default has to have a heavy latest component. We first tried making two calls to the Recommendations API, one with heavy recency bias, but it was hard to get it consisted, so we switched to just splitting the list between the usual Latest algorithm and new recommendations algorithm.
This has the advantage that is preserves some of the “common knowledge” aspect of the current algorithm where you know which posts other people are seeing too, and an author knows that if they get upvoted, their post will be visible automatically and transparently to many people. As discussed elsethread on this post, we want to have a pure-recommendations tab as well and have been waiting on a bit of coding to make that happen.
--
People often have the fear of goodharting on the wrong metric (like clicks) for recommendation algorithms. I think we do need to keep an eye on that, and I want to build more analytics tools for detecting drift here, and more talking to people. I think as we fix up more basic issues like excluding read content and getting it to even recommend posts from older than a year ago[1], we’ll put more attention on is the trend good.
One guess I have is the algorithm is stuck for dumb “structural” reasons, in that it’s been given recent data which is overwhelmingly of people reading recent content, so when it queries “what’s good?” recent content comes out on top even without explicitly training that into the system.