I noticed in Eliezer’s latest MoR update that he now has 18,000 words written, and even when that ‘chapter’ is finished, he still doesn’t plan to post anything, on top of a drought that has now lasted more than half a year.
This doesn’t seem very optimal from the point of view of gaining readers.
But I was wondering how one would quantify that—how one would estimate how many readers Eliezer’s MoR strategy of very rare huge dumps is costing him.
Maybe survivorship curves, where survivorship is defined as ‘posting a review on FF.net’?
So if say during the weekly MoR updates, a reader who posted a review of chapter X posted a review anywhere in X to X+n, that counts as survival of that reader. One problem here is that new readers are constantly arriving… You can’t simply say ‘X readers did not return from chapter 25 to chapter 30, while X+N did not return from chapter 85 to 86, therefore frequent updates are better’ since you would expect the latter to be bigger simply because more people are reading.
And even if you had data on unfavoriting or unfollowing, the important part is the opportunity cost—how many readers would have subscribed with regular updates
If you had total page views, that’d be another thing; you could look at conversions: what percentage of page views resulted in conversions to subscribers for the regular update period versus the feast/fame periods. But I don’t think FF.net provides it and while hpmor.com has Google Analytics, I don’t have access.
Maybe one could look at each chapter pair-wise, and seeing what fraction of reviewers return? The growth might average out since we’re doing so many comparisons… But the delay is now so extreme this would fail: we’d expect a huge growth in reviewers from chapter 85 to chapter 86, for example, simply because it’s been like 8 months now—here too the problem is that the growth in reviewers will be less than what it “should be”. But how do we figure out “should be”?
After some additional discussion with clone_of_saturn, I’ve rejected the idea of survivorship curves; we thought correlations between duration and total review count might work, but interpretation was not clear. So the best current idea is: gather duration between each chapter, the # of reviews posted within X days (where X is something like 2 or 3), plot the points, and then see whether a line fits it better or a logarithm/logistic curve—to see whether growth slowed as the updates were spaced further apart.
Getting the data is the problem. It’s easy to get total reviews for each chapter since FF.net provides them. I don’t know of any way to get total reviews after X days posted, though!
A script or program could probably do it, but I’m not sure I want to bother with all this work (especially since I don’t know how to do Bayesian line or logistic-fitting) if it’s not of interest to anyone or Eliezer would simply ignore any results.
The long drought may cost Eliezer some readers, but the fan base is so enthusiastic that I don’t think it’s a huge loss.
On the other hand, I might be generalizing from myself—I’ve been swamped in good things to read for so long that I’m not extremely focused on the wait for any particular thing.
I noticed in Eliezer’s latest MoR update that he now has 18,000 words written, and even when that ‘chapter’ is finished, he still doesn’t plan to post anything, on top of a drought that has now lasted more than half a year.
This doesn’t seem very optimal from the point of view of gaining readers.
But I was wondering how one would quantify that—how one would estimate how many readers Eliezer’s MoR strategy of very rare huge dumps is costing him.
Maybe survivorship curves, where survivorship is defined as ‘posting a review on FF.net’?
So if say during the weekly MoR updates, a reader who posted a review of chapter X posted a review anywhere in X to X+n, that counts as survival of that reader. One problem here is that new readers are constantly arriving… You can’t simply say ‘X readers did not return from chapter 25 to chapter 30, while X+N did not return from chapter 85 to 86, therefore frequent updates are better’ since you would expect the latter to be bigger simply because more people are reading.
And even if you had data on unfavoriting or unfollowing, the important part is the opportunity cost—how many readers would have subscribed with regular updates
If you had total page views, that’d be another thing; you could look at conversions: what percentage of page views resulted in conversions to subscribers for the regular update period versus the feast/fame periods. But I don’t think FF.net provides it and while hpmor.com has Google Analytics, I don’t have access.
Maybe one could look at each chapter pair-wise, and seeing what fraction of reviewers return? The growth might average out since we’re doing so many comparisons… But the delay is now so extreme this would fail: we’d expect a huge growth in reviewers from chapter 85 to chapter 86, for example, simply because it’s been like 8 months now—here too the problem is that the growth in reviewers will be less than what it “should be”. But how do we figure out “should be”?
After some additional discussion with clone_of_saturn, I’ve rejected the idea of survivorship curves; we thought correlations between duration and total review count might work, but interpretation was not clear. So the best current idea is: gather duration between each chapter, the # of reviews posted within X days (where X is something like 2 or 3), plot the points, and then see whether a line fits it better or a logarithm/logistic curve—to see whether growth slowed as the updates were spaced further apart.
Getting the data is the problem. It’s easy to get total reviews for each chapter since FF.net provides them. I don’t know of any way to get total reviews after X days posted, though!
A script or program could probably do it, but I’m not sure I want to bother with all this work (especially since I don’t know how to do Bayesian line or logistic-fitting) if it’s not of interest to anyone or Eliezer would simply ignore any results.
EDIT: See http://lesswrong.com/r/discussion/lw/fag/analyzing_ffnet_reviews_of_harry_potter_and_the/
The long drought may cost Eliezer some readers, but the fan base is so enthusiastic that I don’t think it’s a huge loss.
On the other hand, I might be generalizing from myself—I’ve been swamped in good things to read for so long that I’m not extremely focused on the wait for any particular thing.