I don’t know; it’s a guess, but it seems a fairly reasonable one. It looks to me as if useless insight-free twaddle tends to get little (or, often, substantially negative) karma.
I’m not claiming this as some sort of exceptionless universal truth, though. Probably carefully tuned insight-free twaddle could get quite a lot of karma.
Um. From experience (I’m very reluctant to give examples, for obvious reasons): Comments that don’t take much thought to come up with seem to net around as much karma as moderately thoughtful ideas, and can be produced in much less time and effort. Of course excellent insights shoot through the roof, but someone who can’t produce many of them in a lifetime can get much more karma pointing out the obvious and making silly jokes than racking their brain. Also, basic ideas in domains LW knows little about gain a lot of karma, which is defensible as comparative advantage, but so do well-phrased restatements of basic ideas on LW.
I’m not claiming that the karma system successfully elicits insight-optimizing behaviour from LW participants (so, in particular, I don’t think anything I’ve said is inconsistent with saying that some people can get more karma from pointing out the obvious than from racking their brain). Only that there’s some correlation between karma and quality and that an intelligent, knowledgeable, wise person will (for any given level of effort and matchedness between their ideas and LW’s collective prejudices and interests) tend to do better than someone less intelligent, knowledgeable and wise.
I still don’t have anything I could reasonably call evidence for this, of course, but if you disagree with it then my casual observations apparently differ from yours.
So what’s a third party to think? Well, obviously they should take your position more seriously than mine because you have a bit more karma than I do. No, wait …
I think it’s very likely to be the case that karma score is strongly correlated with behaviours of which the LW community approves. I think it’s probably the case that rewarded behaviours are more likely to include demonstrating insight, or at least, particular sorts of insight. I think it might be the case that these are correlated with being useful but am not (yet?) convinced.
Fundamentally, it strikes me strongly that it’s very, very hard to assess this empirically. For my day job I’ve been thinking about how you could do a very similar task (for a totally different site) and it’s really very hard without either getting a few known-expert people (who have had no prior contact with the site) to re-rate all the contributions (and hoping for decent inter-rater reliability) or being able to grab a stratified sample of participants and giving them some sort of independent test.
Your good point about carefully tuned insight-free twaddle amused me (thinking of a few examples) and then when I started thinking about how I could tell that apart … rather disturbed me.
It is testable. Costly to test, but testable. If someone wants to test it, they could make a new account, and use it to post lots of useless insight-free twaddle, and see how much karma they get on that account. However, probably the costs of that experiment would be to high to actually pull it off. (Both the cost in time to the tester, and the cost to the signal/noise ratio of Less Wrong of adding that many more posts of useless insight-free twaddle.)
How do you know? (Genuine question.)
I don’t know; it’s a guess, but it seems a fairly reasonable one. It looks to me as if useless insight-free twaddle tends to get little (or, often, substantially negative) karma.
I’m not claiming this as some sort of exceptionless universal truth, though. Probably carefully tuned insight-free twaddle could get quite a lot of karma.
What’s your opinion on this?
Um. From experience (I’m very reluctant to give examples, for obvious reasons): Comments that don’t take much thought to come up with seem to net around as much karma as moderately thoughtful ideas, and can be produced in much less time and effort. Of course excellent insights shoot through the roof, but someone who can’t produce many of them in a lifetime can get much more karma pointing out the obvious and making silly jokes than racking their brain. Also, basic ideas in domains LW knows little about gain a lot of karma, which is defensible as comparative advantage, but so do well-phrased restatements of basic ideas on LW.
I’m not claiming that the karma system successfully elicits insight-optimizing behaviour from LW participants (so, in particular, I don’t think anything I’ve said is inconsistent with saying that some people can get more karma from pointing out the obvious than from racking their brain). Only that there’s some correlation between karma and quality and that an intelligent, knowledgeable, wise person will (for any given level of effort and matchedness between their ideas and LW’s collective prejudices and interests) tend to do better than someone less intelligent, knowledgeable and wise.
I still don’t have anything I could reasonably call evidence for this, of course, but if you disagree with it then my casual observations apparently differ from yours.
So what’s a third party to think? Well, obviously they should take your position more seriously than mine because you have a bit more karma than I do. No, wait …
I think it’s very likely to be the case that karma score is strongly correlated with behaviours of which the LW community approves. I think it’s probably the case that rewarded behaviours are more likely to include demonstrating insight, or at least, particular sorts of insight. I think it might be the case that these are correlated with being useful but am not (yet?) convinced.
Fundamentally, it strikes me strongly that it’s very, very hard to assess this empirically. For my day job I’ve been thinking about how you could do a very similar task (for a totally different site) and it’s really very hard without either getting a few known-expert people (who have had no prior contact with the site) to re-rate all the contributions (and hoping for decent inter-rater reliability) or being able to grab a stratified sample of participants and giving them some sort of independent test.
Your good point about carefully tuned insight-free twaddle amused me (thinking of a few examples) and then when I started thinking about how I could tell that apart … rather disturbed me.
It is testable. Costly to test, but testable. If someone wants to test it, they could make a new account, and use it to post lots of useless insight-free twaddle, and see how much karma they get on that account. However, probably the costs of that experiment would be to high to actually pull it off. (Both the cost in time to the tester, and the cost to the signal/noise ratio of Less Wrong of adding that many more posts of useless insight-free twaddle.)