I think it’s very likely to be the case that karma score is strongly correlated with behaviours of which the LW community approves. I think it’s probably the case that rewarded behaviours are more likely to include demonstrating insight, or at least, particular sorts of insight. I think it might be the case that these are correlated with being useful but am not (yet?) convinced.
Fundamentally, it strikes me strongly that it’s very, very hard to assess this empirically. For my day job I’ve been thinking about how you could do a very similar task (for a totally different site) and it’s really very hard without either getting a few known-expert people (who have had no prior contact with the site) to re-rate all the contributions (and hoping for decent inter-rater reliability) or being able to grab a stratified sample of participants and giving them some sort of independent test.
Your good point about carefully tuned insight-free twaddle amused me (thinking of a few examples) and then when I started thinking about how I could tell that apart … rather disturbed me.
I think it’s very likely to be the case that karma score is strongly correlated with behaviours of which the LW community approves. I think it’s probably the case that rewarded behaviours are more likely to include demonstrating insight, or at least, particular sorts of insight. I think it might be the case that these are correlated with being useful but am not (yet?) convinced.
Fundamentally, it strikes me strongly that it’s very, very hard to assess this empirically. For my day job I’ve been thinking about how you could do a very similar task (for a totally different site) and it’s really very hard without either getting a few known-expert people (who have had no prior contact with the site) to re-rate all the contributions (and hoping for decent inter-rater reliability) or being able to grab a stratified sample of participants and giving them some sort of independent test.
Your good point about carefully tuned insight-free twaddle amused me (thinking of a few examples) and then when I started thinking about how I could tell that apart … rather disturbed me.