Whatever you [Luke] were split testing for (a quick look suggests “Lesson” vs. “Rationality skill”) is probably undone by the first reply comments on this post compared to the other one.
An interesting observation that was noted at Hacker News a while back is that the top rated comment on almost any opinion piece is disagreement—because people who passionately disagree are more likely to look for an argument to back in the comments.
If you skim discussion sites where voting moves comments up and a culture of dissent being respected reigns—you’ll see it’s usually true.
But the difference between the A version and the B version is that, as of the time of this writing, B starts with “I downvoted because...” whereas A’s first comment is also disagreement, but of a more encouraging sort. I think this will probably dominate the results far more than the phrasing and exact structure of lesson/skills learned.
The obvious solution is then to hide comments until a user has voted on an article. Perhaps with a third option to abstain instead of up- or downvoting.
You could also test this effect with a similar AB test. Just give the two groups of readers the exact same article, but with a different first comment added by a collaborator or sock puppet.
Whatever you [Luke] were split testing for (a quick look suggests “Lesson” vs. “Rationality skill”) is probably undone by the first reply comments on this post compared to the other one.
An interesting observation that was noted at Hacker News a while back is that the top rated comment on almost any opinion piece is disagreement—because people who passionately disagree are more likely to look for an argument to back in the comments.
If you skim discussion sites where voting moves comments up and a culture of dissent being respected reigns—you’ll see it’s usually true.
But the difference between the A version and the B version is that, as of the time of this writing, B starts with “I downvoted because...” whereas A’s first comment is also disagreement, but of a more encouraging sort. I think this will probably dominate the results far more than the phrasing and exact structure of lesson/skills learned.
The obvious solution is then to hide comments until a user has voted on an article. Perhaps with a third option to abstain instead of up- or downvoting.
You could also test this effect with a similar AB test. Just give the two groups of readers the exact same article, but with a different first comment added by a collaborator or sock puppet.