Care to translate that “should” into a well-specified forecast with attached probability? :)
We are assuming that the change will affect comments at −4 or lower, but it might not change the number of large discussions under comments at −3. There might be discussions that transition between censored and uncensored. The censorship might actually prevent comments at −4 from being downvoted even worse, and thus could perversely make transitions back to −3 more likely.
It would be interesting to run some stats on how frequent the event of interest is (assuming we can specify it coherently), before and after the change. Based on my memory of the LW codebase, votes are stored transactionally, so it should be possible to compute before/after statistics at any time.
There might be discussions that transition between censored and uncensored.
This could be annoying, if we have to check all the comments upstream from the one we’re responding to, to make sure there isn’t a comment that might be downvoted to −4 in the future and make the effort a waste. I pointed out a bunch of potential downsides of this proposal to Eliezer but even I didn’t think of this one.
Yeah, most of the systems that do this sort of thing seem to hide the low-scoring comments but show high-voted children, avoiding that sort of problem.
That’s not necessarily what I mean by a “well-specified forecast”. Be careful not to confuse “precise” and “accurate”… For instance by “downvoted” do you mean “net votes below 0”, “having received any downvotes”, or “net votes at −4 or below”?
The last—by “downvoted threads” I meant “threads descended from a comment at −4 or less” (though there should also be an effect for threads whose most downvoted parent is at −3 or at −2). It’s a bit of a pity there isn’t a standard name for those.
OK, so to be clear: you’re predicting a roughly 50% decrease of the population “comments which are descendants of comments downvoted −4 or more”. This at “I’d be pretty surprised if it turned out otherwise”, which is my verbal equivalent of 70%. (For 80% it’s “I would be shocked” and for 90% it’s “I’d seriously question my worldview on the topic in question”, for 99% it’s “you should really not be messing around with anything remotely connected with that topic, you’re dangerous to yourself and others”.)
Here are some of the uncertainties.
We don’t know how large this population is currently. There is a subjective feeling that this number is significant and annoyingly so, but if it is small then it may be hard to detect an effect among the noise.
We don’t know many new comments arise from replies to Recent Comments, as opposed to two people going back and forth, or people explicitly looking for new stuff in a discussion they’re following, or people following a particular commenter.
We don’t know how fast low-quality comments get to −4 before they have accrued substantial discussion, or alternately the ratio between number of comments accumulated before getting to −4 and comments accumulated after.
Sadly, I’m about 65% sure that we’ll never get to have actual stats on the above, or on the prediction itself. :-/
Care to translate that “should” into a well-specified forecast with attached probability? :)
We are assuming that the change will affect comments at −4 or lower, but it might not change the number of large discussions under comments at −3. There might be discussions that transition between censored and uncensored. The censorship might actually prevent comments at −4 from being downvoted even worse, and thus could perversely make transitions back to −3 more likely.
It would be interesting to run some stats on how frequent the event of interest is (assuming we can specify it coherently), before and after the change. Based on my memory of the LW codebase, votes are stored transactionally, so it should be possible to compute before/after statistics at any time.
This could be annoying, if we have to check all the comments upstream from the one we’re responding to, to make sure there isn’t a comment that might be downvoted to −4 in the future and make the effort a waste. I pointed out a bunch of potential downsides of this proposal to Eliezer but even I didn’t think of this one.
Yeah, most of the systems that do this sort of thing seem to hide the low-scoring comments but show high-voted children, avoiding that sort of problem.
Very approximately, I’d say I expect at least 50% less comments posted in downvoted threads, with probability 70%.
(though I don’t think that adding precise numbers adds much to the discussion)
That’s not necessarily what I mean by a “well-specified forecast”. Be careful not to confuse “precise” and “accurate”… For instance by “downvoted” do you mean “net votes below 0”, “having received any downvotes”, or “net votes at −4 or below”?
The last—by “downvoted threads” I meant “threads descended from a comment at −4 or less” (though there should also be an effect for threads whose most downvoted parent is at −3 or at −2). It’s a bit of a pity there isn’t a standard name for those.
OK, so to be clear: you’re predicting a roughly 50% decrease of the population “comments which are descendants of comments downvoted −4 or more”. This at “I’d be pretty surprised if it turned out otherwise”, which is my verbal equivalent of 70%. (For 80% it’s “I would be shocked” and for 90% it’s “I’d seriously question my worldview on the topic in question”, for 99% it’s “you should really not be messing around with anything remotely connected with that topic, you’re dangerous to yourself and others”.)
Here are some of the uncertainties.
We don’t know how large this population is currently. There is a subjective feeling that this number is significant and annoyingly so, but if it is small then it may be hard to detect an effect among the noise.
We don’t know many new comments arise from replies to Recent Comments, as opposed to two people going back and forth, or people explicitly looking for new stuff in a discussion they’re following, or people following a particular commenter.
We don’t know how fast low-quality comments get to −4 before they have accrued substantial discussion, or alternately the ratio between number of comments accumulated before getting to −4 and comments accumulated after.
Sadly, I’m about 65% sure that we’ll never get to have actual stats on the above, or on the prediction itself. :-/