Data Analysis of LW: Activity Levels + Age Distribution of User Accounts
Epistemic: I rarely trust other people’s data analysis, I only half trust my own. Right now, analytics is only getting a slice of my attention and this work is not as thorough as I’d like, but I think the broad strokes picture is correct. I have probably failed to include enough clarifications and disclaimers on where we should expect the data to be inaccurate. Feedback on my approach welcome.
I’ve been doing some analytics work for the LessWrong 2.0 team since September last year (since March I’ve been doing other work too, but that’s not relevant here). This post will hopefully be the first a series which will eliminate the backlog of analytics results I’ve been wanting to share.
This post is probably not the ideal starting point—that would be probably be a big picture general overview of LessWrong usage since the beginning—but it is some of my most recent work and therefore is easiest to share. Still, it does show things about the bigger picture.
Warning: The graphs are repetitive even though they’re showing different things. I’ve included them all for completeness, but you can just read my summary/interpretations while looking at only some of them.
Distribution of User Account Age
Question: LW2 seems to be doing well, but is that just because we’re retaining/re-engaging a devoted base of older users despite not signing up new users?
Answer: Activity on LW2 is coming from both new and old users across all activity types (posts, comments, votes, and views). The project is succeeding at getting new people to create accounts and engage.
In fact, there have consistently been more new users voting and viewing each month since LW2 launched than throughout LW’s past. The number of new users posting each month is roughly the same as historical levels. The number of new users who are commenting has declined (though the percentage new users is roughly the same), however this is consistent with the trend that comment volume on LW2 has not recovered from The Great Decline of 2015-2017 the way other metrics have.
Meaning of the Graphs
I plotted graphs for each activity type (posts, comments, votes, and views) and the corresponding population of users which engages in those activities. For each user engaging in each activity type, I calculated the “age” of that account since it first engaged in that activity type, i.e. in the graph for users who post, the age of the user account in a given month is the number of days elapsed since that account first posted. In the graph for commenters, is the days elapsed since the account first commented. This avoids certain complications which inconsistencies in how the data was recorded for different activities over LessWrong’s histories.
I segmented the user accounts into four “buckets” based on their “age” [since first engaging in activity of type X].
0 − 90 days
90 − 360 days (~3 months to 12 months old)
360 − 720 days (~1 to 2 years)
720 − 10,000 days (~2+ years )
When I’ve said new user accounts, I have been meaning 0 − 90 days; when I’ve said old users, I’ve meant the 720+ days bucket.
Caveat: we believe that many old users created new accounts when LW2 launched and this is somewhat confounding the data, though not necessarily a lot.
Reading the Graphs
X-Axis is time
Values plotted are the total values for each month
Y-Axis is about the number of individuals engaging in a behavior in a given month, e.g there ~600 people who viewed posts, 30% of which have had less 90 days elapsed since they were first recorded viewing a post as a logged-in user***.
In each set of graphs by activity:
The first set (2x2) shows a time series line for each age bucket segment alone.
The second long graph shows an area plot time series with age buckets segments stacked. This lets you see overall size of the population engaging an activity type over time.
The third graph is a 100% area plot which shows the composition of the overall population by “age” of the user accounts over time.
A moving-average filter of three time months has been applied for smoothing.
***All data here is from logged-in users!!! Including views. View counts of non-logged in users are over an order of magnitude higher.
Poster Distribution of Age
Questions and posts with 2 or less upvotes have been filtered out. Event/meetup posts have not.
In addition to our primary focus on the age distribution of accounts, we can note the inflection point occurring in September/October 2017. This corresponds to the launch of the LessWrong 2.0 Open Beta 9-20 and publishing of Eliezer’s Inadequate Equilibria on LW* on 10-28. I have marked 2017-10-01 on the graphs with a dotted black line.
*The LW2 team requested Inadequate Equilibria be published on LW2 as an initial draw.
We see from these graphs that the number of users making posts each month is almost as high (~75%) as historical levels, especially those after 2013.
Unsurprisingly, over time more profiles fall into in the “2+ year since their first post” bucket since the longer LW has existed, the more profiles which are at 2+ years can exist. Percentage of users posting with accounts less than 90 days since first post (this includes their first post) has remained almost constant over time with the exception of during the decline period 2015-2017.
A small aside: it’s interesting to note that the nature of posts has shifted somewhat. The median post length on LW2 (~1000 words) is double that from old LW (~500 words). Main posts were on average much longer than Discussion posts (median ~1000 words vs ~300 words). The distribution of post length on LW2 almost exactly matches that LW’s Main section despite having far more posts. The net result is that at least many total words of posts are being written on LW2 compared to legacy LW.
Inserting some analysis from a few months ago. I haven’t re-checked this before including though, so slightly higher chance that I messed something up.
Post Length Distributions for LW1 Discussion, LW1 Main, and LW2
Word count is naively calculated as character count divided by 6, hence the fractional values.
I vaguely suspect that the shift in post length signifies a change in how LessWrong is used and that this is related to the large reduction in comment volume (see next section). A hypothesis is that old LW used to be used for some of the same uses as Facebook and other social media currently fulfils for people, and that new LW2 is now primarily serving some other need.
Commenter Distribution of Age
The graphs for commenters reveal a significant reality for LW2: while post, vote, and view have resurged since The Great Decline of 2015-2017, commenting levels have not returned to anything near historical levels. Since the LW2.0 launch, the percentage of commenters who are new commenters are at its highest levels since 2013 while commenters who began commenting 2+ years ago has been steady at 50% of commenters. The topmost left graph (blue line) shows that there were no new users commenting in the period before LW2 but that this changed with the launch of LW2 and Inadequate Equlibria.
Voter Distribution of Age
The graphs for population of voters tell an interesting tale. There has been a dramatic increase in the number of new users voting while the number and proportion of accounts who first voted 2+ years ago has stayed almost steady/declined a little. The net result is that voters who first voted within the last two years are making up 60% of the voting population. (The effects on overall karma distributed can’t be straightforwardly inferred from this alone since it will depend on how many votes each user makes and their karma scores.)
Logged-In Viewer Distribution of Age
The distribution of user account age for logged-in viewers is similar to that for votes. Large uptick for new accounts, yet no growth among older user accounts. The data here however is “compromised” since in March 2018 all users were logged-out. Users who failed to login again (which is unnecessary if you are not posting, commenting, or voting), would no longer be detected. The drop in logged-in viewer population can be seen in early 2018, particularly in the 2+ year plus time series (red line). After that point, it is mostly flat similar to the case for voters.
Concluding Thoughts
It’s heartening to see that LW2 has made a difference to the trajectory of LW. A site which was nearly put into read-only archive mode is definitely alive and kicking. Counter to my fears, LW2 is drawing in new users rather than purely being sustained by a committed core of older users from LW1. This is despite not yet focusing on recruiting new users, e.g. via promotion of content on social media.
However, the rate of new users which come on is matched by the number of users failing to return (be they older users or new users who aren’t sticking around). Overall, most of the straightforward analytics metrics for LW have not grown significantly since its launch. I suspect that if we understand what is going on with retention, we might be able to hold onto more of the new users and actually cause upwards growth. . . . assuming we want that. I and others on the team don’t blindly believe that growth for the sake of growth is good. We’ll continue to think carefully about any actions we might take that even if they caused “growth”, might cause LW2 not to be the place we want it to be.
I’ve only had a very cursory look at retention. I found that new users were returning in the first month after signing up at historical levels (~30%), but that retention three months after joining is less than half of historical levels (~20%->%5). This is odd. However, these numbers are only from a cursory glance and I haven’t been very thorough yet either in coding mistakes or even thinking about it right.. This paragraph is low confidence.
Another point is that users of LW2 don’t use the site, on average, as much as they did during LW’s peak. Up to 2014, the median user was present on LW for 4-5 days each month (i.e. 4⁄31 days); in the last couple of years that has been 2-3 days. This might correspond to fewer people commenting since being engaged in ongoing comment threads might keep people coming back. The team is curious how a revamped email notification system (currently under development) will affect frequency of visiting LW.
Lastly, and I think it’s okay for me to say this, is that many of the most significant contributors to LW in the past are still present on the site—lurking—even if they post and comment far less. (I will soon write a post on how we use user data for analytics and decision-making; rest assured we have an extremely hard policy against ever sharing individual user data which is not public.) I think it’s a good sign that LW2 is generating enough content and discussion that these users still want to keep up date with LW2. It makes me hopeful that LW2 might even become a (the?) central place of discussion.
For those interested in working with LessWrong data
To protect user privacy, we’re not able to grant full-access to our database to the public, however there might be more limited datasets which we can release. If there’s enough interest, I’ll discuss this with the team.
Here’s a pessimistic takeaway. One should expect a site that was providing a lot of value to its users to grow, even if it wasn’t explicitly trying to.
Perhaps most of the value of the site is in the fact that it has posts, comments and votes. Beyond that it’s the value of the content, and that is modest and static.
View is lightly held, and I’m not drawing much in the way of conclusions from it. I am still impressed with the feature roadmap and the velocity of improvement.
Yes, I do expect if you’re generating enough value you will should see automatic growth, from which I infer that LessWrong 2.0 isn’t providing that much value to its users right now. Though I think there’s a mix of reasons to not be especially pessimistic:
Successful companies with worthwhile products seem to me to still have to invest in getting new users. My feeling (not really backed by data) is that you have to be outstandingly good to get full-on organic growth without trying. Not being there doesn’t mean you’re not providing value.
We see in the graphs that LW was not growing for most of its history: most of the metrics peak around 2011 and remain steady or decline slowly until 2015. I would argue that despite not growing, LW was still providing a lot of value to its users and the world during this period.
My outside view and inside view lead me believe hockey stick growth to be real. Part of my model is that even if you’re doing many things right, it might require having all the pieces click into place before dramatic growth starts. The pieces are connected in series, not parallel. Relatedly, sometimes the key to winning big is just not dying for long enough.
LW2 is much more fussy about which value we provide to which users than I expect most companies are. Most companies are trying to find approximately any product and corresponding set of users such that the value provided to users can be used to extract money somehow. In contrast, I care only about finding products and users to whom providing value will generate significant value for the world at large (particularly through the development/training of rationality and general intellectual progress on important problems). I think this is a much more restrictive constraint. It leads me (and I think the team generally) to want to forego many opportunities for user/activity growth if we don’t think they’ll lead to greater value for the world overall. Because of this, I’m not worried yet that we haven’t hit on a formula for providing value that’s organically getting a lot of growth. We have a narrow target.
Generally, I (and others on the team) don’t consider LW to have achieved the nonprofit analog of product-market fit. More precisely, we haven’t hit upon a definite and scalable mechanism for generating large amounts of value for the world (especially intellectual progress). I have an upcoming post I wish I could link to which describes various ideas we’re trying or thinking about as mechanisms. Open Questions is one such attempt.
I’m unsure of your meaning here. Are you saying there’s content separate from posts and comments? I consider all our content to fall into those categories. Some of it is arguably static, but I’m not sure I’d say modest? Can you say more what you meant by that?
Paul Graham is such a fun read, but when I have my skeptical hat on I don’t find myself convinced. What’s the mechanism of action? If LW doesn’t die it will eventually achieve its aims because .. ?
I do like the lens of product-market fit, and I tentatively agree LW doesn’t really have it. I guess you could say that you have successfully avoided dying and now you get to continuing swinging at ideas until you hit a home run.
I think you really answered it, as long as you don’t die:
Of course, this works a lot better if you’re learning as you go and your successive attempts fall closer to success than if you’re randomly trying things. I’d like to think that the LW isn’t randomly swinging. There are also hard pieces like keeping the team engaged and optimistic even as things don’t seem dramatically different (though you could call that part of not dying).
Here is one framing: If you have a site that has high user turnover, while also having a stable number of users, then if you increase your retention by just a tiny bit, you are now in the dimension of exponential growth (and if your retention declines, you are now in the domain of exponential decline). It actually feels weird to have as high user turnover and overall visitors as we have, with some significant retention, but in a way that is almost perfectly cancelled out by people leaving (I have some thoughts on explanations of this that I might write up some other day).
Clarifying:
The hypothesis is: the value of the site technical functionality to the average user is mostly the posts, comments and votes. That’s already built so the work on the margin hasn’t increased the value that much. The real room for variation is the value of the content (writing) on the site. The value of that content is modest (not huge) and static (not growing).
Is there a reason that post view count is not public? (for each post? Anonymous counter)
Old style mybb forums had this function. Seems simple and easy to implement, OTOH at least have the post author be able to know how many views they are getting?
I’d be interested in the views a post is getting, the total site views a month, and therefore the relevance to the user base of my posts. (independent of comment and vote metric data)
It’s definitely an intentional decision, though I am open to discussing whether it’s a good one. We have an admin-only view-counter, and what posts tend to get a lot of views is quite uncorrelated, and in some domains pretty predictably anti-correlated with what I think good content on LessWrong looks like.
I am worried that if we add a prominent view-counter, we will start a goodharting process on a metric that the whole internet is already optimizing on (since indeed, the posts that get the most views are just the ones that are closest to broad internet clickbait and/or community drama).
For me, I don’t write a clickbait and I don’t write a community drama. But I’ve written posts with 5 hours of work and posts with 30mins of work. And different styles and qualities of 30min posts. I’d love to know if people are reading them.
A post with 100 views and +10 up votes VS a post with 15 views and +10 up votes is a very different thing.
I certainly can imagine local improvements people would get from having more information here, the question is whether you can implement the function without causing all the longterm distortions Habryka described.
An option is giving the feature to people with high karma so you have to demonstrate some acculturation before being handed Goodhart’s Key, but honestly I’m not sure there are people, high karma or otherwise, who I really trust to remain impervious to the subtle pressure to write more clickbaity things, over time.
There are two features. “author sees view count” and “public sees view count”. Which one are you talking about?
Author sees viewcount
If it’s a non-public view count, I don’t see it becoming a goodheart metric. If something is too clickbait or trash, it would get downvotes. If it doesn’t get downvotes, maybe there’s good reasons.
Maybe it would be worth internally having:
page view count
upvote count
downvote count
vote total (also possibly up and down vote total)
comment count
some sort of relative metric that can compare this article to the other articles nearby.