Good point, here is why I do think the current implementation is probably worth the cost:
We’ve talked to a bunch of core contributors and people who are now doing great work in AI alignment and other domains, and a pretty surprising fraction of them said that they got engaged because of the karma leaderboard on the old LessWrong. Scott Garrabrant was one of the top people here, who mentioned that early on the karma system was quite important to keep him motivated and to continue working on AI Alignment.
We had the feature activated for mods only in the last three weeks, and I did notice it generally increasing my satisfaction with LessWrong by a pretty significant amount. I have it set to daily-batching, and it makes me quite happy to see when some old comments of mine are getting upvoted, and generally makes me feel more appreciated and motivated in a bunch of important ways. I’ve found it particularly exciting to see comments and posts of mine that are many months old still getting upvotes, which I think makes me generally better calibrated on the long-term value of writing things up and making them public, instead of just talking to people in-person which tends to have a higher immediate reward but a lower long-term reward.
I actually think it is quite valuable for users to be broadly responsive to upvotes and downvotes, and in particular want to give users more detailed analytics on when their stuff gets upvoted. Another part of the karma notification system that we are currently working on is to add a page with a time-range slider that allows you to run basic analytics on when and how your stuff got upvoted and downvoted. For many early users, being responsive to general karma I think is a good way to improve their writing and ideas, much better than click-count which I think is the standard metric that people would use if they don’t have something else available.
We had the feature activated for mods only in the last three weeks, and I did notice it generally increasing my satisfaction with LessWrong by a pretty significant amount.
What’s the current thinking on the way it displays downvotes? My today’s karma change showed a red −7, and while I’ve got enough karma that I don’t care that much about karma changes in general and could just shrug it off, it did feel more distinctly more unpleasant than much larger karma increases in the previous days have felt pleasant. I’m a little worried that this might make new users take downvotes even more harshly than they otherwise would.
Sites like FB tend to basically only display positive information, probably for this reason—e.g. they tell you when someone accepts your friend request or likes your comment, but they don’t tell you when someone unfriends you or (AFAIK) when someone deletes your comment. I’m not sure if that’s quite the right answer here—you did mention wanting to make users also more responsive to downvotes—but having a user come back to the site and run into a red negative number as one of the first things they see feels a little harsh. Maybe have the immediately-visible-number only display the sum of upvotes, but still show the downvotes when clicked on?
Hmm, I was worried about that, but never ended up having a negative karma day during the whole test period (probably because I just wrote a bunch of pretty straightforward meta comments).
I do find myself somewhat surprised that you ended up with a net negative balance, though statistically that is bound to happen sometimes. I can imagine that we should probably just display those in grey and not in red, since I think people naturally respond a lot stronger to negative feedback than positive feedback. But that might still not be enough. I can imagine not showing negative values and instead just showing a grey “<0” or something like that. I will experiment a bit.
One important question I have is “does anyone want to see their negative karma updates?” (I personally don’t have a particular desire to see it, and would want to make sure there’s at least a few existence proofs of the feature being useful to someone)
It’s plausibly correct to provide the option, so long as it isn’t the default. (Options: Show all, show none, show only positive, show only negative. The last option being something that no one should ever use, provided only for symmetry.)
The main thing I like about the ‘only downvotes’ option is that it’s kind of funny and pointless. This suits my aesthetic. I could imagine trying it out for a few weeks to see what happens / to call the bluff of the part of my primate brain that thinks social disapproval from strangers is an x-risk. :)
The main thing I like about the ‘only downvotes’ option is that it’s kind of funny and pointless.
I feel the same about the ‘only upvotes’ option. Applying the reversal test, imagine that most people treat the ‘only downvotes’ option seriously and suggest that it should be the default, since it agrees with the usual norms of in-person conversation. Downvotes could even measure popularity if there was enough volume, in the meantime the sum of absolute values of upvotes and downvotes can play that role.
Applying the reversal test, imagine that most people treat the ‘only downvotes’ option seriously and suggest that it should be the default, since it agrees with the usual norms of in-person conversation.
The usual conversation norms are “only say positive things, unless you think it’s really important to give negative feedback.” This maps to “people receive messages of positive reinforcement, generally don’t receive messages of significant negative reinforcement unless it’s really important, and therefore treat all negative reinforcement as really important.”
One could argue that (in both regular conversation and also lesswrong) the role should be reversed, wherein people usually only give positive reinforcement if it’s really important. But, that’d be a change from the conversational status quo, not in agreement with it.
The reversal test is with respect to the norm, not with respect to ways of handling a fixed norm. So imagine that the norm is the opposite, and see what will happen. People will invent weird things like gaging popularity based on number of downvotes, or sum of absolute values of upvotes and downvotes, when there are not enough downvotes. This will work about as well as what happens with the present norm. In that context, the option of “only upvotes” looks funny and pointless, but we can see that it actually isn’t, because we can look from the point of view of both possible norms.
When an argument goes through in the world of the opposite status quo, we can transport it to our world. In this case, we obtain the argument that “only downvotes” is not particularly funny and pointless, instead it’s about as serviceable (or about as funny and pointless) as “only upvotes”, and both are not very good.
If I’m having lunch with a friend, then my usual expectation is that I’ll get strong compliments if they adore my clothing style, but I won’t get strong criticisms if they strongly dislike it, unless I explicitly opt in to receiving the latter feedback. Most people seem to treat high-salience personal compliments as opt-out, while treating high-salience personal criticisms as opt-in. This can be outweighed if the criticism is important enough, but otherwise, criticism tends to be relatively mild and cloaked in humor or indirection.
Thinking about it in those terms, it makes sense to me to treat upvotes as similar to “person says they love my haircut” and downvotes as similar to “person says they hate my haircut.” I probably want to be able to view both kinds of feedback in a time and place of my choosing, but I don’t want to have the latter feedback tossed my way literally every time I open Chrome or check my email.
It might be that those norms are fine for personal style, but that we want to promote better, more pro-criticism norms in areas that matter more. We might want to push in the direction of making critical feedback opt-out, so people can (a) update faster on things that do matter a lot, and (b) perhaps get some useful exposure therapy that will make us better at receiving tips, pushback, and contrary views in the future. Mostly I’m just making this comment so folks feel comfortable talking about their preferences openly, without feeling like they’re Bad Rationalists if they’re not already convinced that it’s useful for them personally to receive a regular stream of downvote notifications (in the world where they get a lot of downvotes).
I think I might have posted significantly less on the original Less Wrong if it had made downvotes this salient at a time when I wasn’t yet confident in the quality of my contributions, and if it didn’t offer a “don’t show downvotes” option.
I’ve found it particularly exciting to see comments and posts of mine that are many months old still getting upvotes, which I think makes me generally better calibrated on the long-term value of writing things up and making them public, instead of just talking to people in-person which tends to have a higher immediate reward but a lower long-term reward.
Even vote counts underestimate viewership numbers pretty drastically, don’t they? I remember making comments with embedded polls where the poll got 100+ votes and the comment was sitting at +2. (And only logged-in users can vote in polls!)
Good point, here is why I do think the current implementation is probably worth the cost:
We’ve talked to a bunch of core contributors and people who are now doing great work in AI alignment and other domains, and a pretty surprising fraction of them said that they got engaged because of the karma leaderboard on the old LessWrong. Scott Garrabrant was one of the top people here, who mentioned that early on the karma system was quite important to keep him motivated and to continue working on AI Alignment.
We had the feature activated for mods only in the last three weeks, and I did notice it generally increasing my satisfaction with LessWrong by a pretty significant amount. I have it set to daily-batching, and it makes me quite happy to see when some old comments of mine are getting upvoted, and generally makes me feel more appreciated and motivated in a bunch of important ways. I’ve found it particularly exciting to see comments and posts of mine that are many months old still getting upvotes, which I think makes me generally better calibrated on the long-term value of writing things up and making them public, instead of just talking to people in-person which tends to have a higher immediate reward but a lower long-term reward.
I actually think it is quite valuable for users to be broadly responsive to upvotes and downvotes, and in particular want to give users more detailed analytics on when their stuff gets upvoted. Another part of the karma notification system that we are currently working on is to add a page with a time-range slider that allows you to run basic analytics on when and how your stuff got upvoted and downvoted. For many early users, being responsive to general karma I think is a good way to improve their writing and ideas, much better than click-count which I think is the standard metric that people would use if they don’t have something else available.
What’s the current thinking on the way it displays downvotes? My today’s karma change showed a red −7, and while I’ve got enough karma that I don’t care that much about karma changes in general and could just shrug it off, it did feel more distinctly more unpleasant than much larger karma increases in the previous days have felt pleasant. I’m a little worried that this might make new users take downvotes even more harshly than they otherwise would.
Sites like FB tend to basically only display positive information, probably for this reason—e.g. they tell you when someone accepts your friend request or likes your comment, but they don’t tell you when someone unfriends you or (AFAIK) when someone deletes your comment. I’m not sure if that’s quite the right answer here—you did mention wanting to make users also more responsive to downvotes—but having a user come back to the site and run into a red negative number as one of the first things they see feels a little harsh. Maybe have the immediately-visible-number only display the sum of upvotes, but still show the downvotes when clicked on?
One mild tweak could just be to not make it “red”
Hmm, I was worried about that, but never ended up having a negative karma day during the whole test period (probably because I just wrote a bunch of pretty straightforward meta comments).
I do find myself somewhat surprised that you ended up with a net negative balance, though statistically that is bound to happen sometimes. I can imagine that we should probably just display those in grey and not in red, since I think people naturally respond a lot stronger to negative feedback than positive feedback. But that might still not be enough. I can imagine not showing negative values and instead just showing a grey “<0” or something like that. I will experiment a bit.
One important question I have is “does anyone want to see their negative karma updates?” (I personally don’t have a particular desire to see it, and would want to make sure there’s at least a few existence proofs of the feature being useful to someone)
I don’t like the idea that LW will tell me my daily karma change but only if it’s good news.
I would also feel somewhat uncomfortable with this.
It’s plausibly correct to provide the option, so long as it isn’t the default. (Options: Show all, show none, show only positive, show only negative. The last option being something that no one should ever use, provided only for symmetry.)
I personally wouldn’t bother including the last option, symmetry be damned. Choices are bad.
The main thing I like about the ‘only downvotes’ option is that it’s kind of funny and pointless. This suits my aesthetic. I could imagine trying it out for a few weeks to see what happens / to call the bluff of the part of my primate brain that thinks social disapproval from strangers is an x-risk. :)
I feel the same about the ‘only upvotes’ option. Applying the reversal test, imagine that most people treat the ‘only downvotes’ option seriously and suggest that it should be the default, since it agrees with the usual norms of in-person conversation. Downvotes could even measure popularity if there was enough volume, in the meantime the sum of absolute values of upvotes and downvotes can play that role.
I’m not sure I parse this comment.
The usual conversation norms are “only say positive things, unless you think it’s really important to give negative feedback.” This maps to “people receive messages of positive reinforcement, generally don’t receive messages of significant negative reinforcement unless it’s really important, and therefore treat all negative reinforcement as really important.”
One could argue that (in both regular conversation and also lesswrong) the role should be reversed, wherein people usually only give positive reinforcement if it’s really important. But, that’d be a change from the conversational status quo, not in agreement with it.
The reversal test is with respect to the norm, not with respect to ways of handling a fixed norm. So imagine that the norm is the opposite, and see what will happen. People will invent weird things like gaging popularity based on number of downvotes, or sum of absolute values of upvotes and downvotes, when there are not enough downvotes. This will work about as well as what happens with the present norm. In that context, the option of “only upvotes” looks funny and pointless, but we can see that it actually isn’t, because we can look from the point of view of both possible norms.
When an argument goes through in the world of the opposite status quo, we can transport it to our world. In this case, we obtain the argument that “only downvotes” is not particularly funny and pointless, instead it’s about as serviceable (or about as funny and pointless) as “only upvotes”, and both are not very good.
If I’m having lunch with a friend, then my usual expectation is that I’ll get strong compliments if they adore my clothing style, but I won’t get strong criticisms if they strongly dislike it, unless I explicitly opt in to receiving the latter feedback. Most people seem to treat high-salience personal compliments as opt-out, while treating high-salience personal criticisms as opt-in. This can be outweighed if the criticism is important enough, but otherwise, criticism tends to be relatively mild and cloaked in humor or indirection.
Thinking about it in those terms, it makes sense to me to treat upvotes as similar to “person says they love my haircut” and downvotes as similar to “person says they hate my haircut.” I probably want to be able to view both kinds of feedback in a time and place of my choosing, but I don’t want to have the latter feedback tossed my way literally every time I open Chrome or check my email.
It might be that those norms are fine for personal style, but that we want to promote better, more pro-criticism norms in areas that matter more. We might want to push in the direction of making critical feedback opt-out, so people can (a) update faster on things that do matter a lot, and (b) perhaps get some useful exposure therapy that will make us better at receiving tips, pushback, and contrary views in the future. Mostly I’m just making this comment so folks feel comfortable talking about their preferences openly, without feeling like they’re Bad Rationalists if they’re not already convinced that it’s useful for them personally to receive a regular stream of downvote notifications (in the world where they get a lot of downvotes).
I think I might have posted significantly less on the original Less Wrong if it had made downvotes this salient at a time when I wasn’t yet confident in the quality of my contributions, and if it didn’t offer a “don’t show downvotes” option.
Even vote counts underestimate viewership numbers pretty drastically, don’t they? I remember making comments with embedded polls where the poll got 100+ votes and the comment was sitting at +2. (And only logged-in users can vote in polls!)
FYI, I talked to Oliver about this and he says:
The average post gets between 200 and 500 unique views in the first month, with curated ones usually getting around 2k to 5k.
Usually viewership appears to be roughly a factor 20 or 30 times the vote count.