I want to suggest having new posts/comments be normal upvoted by default (by the author) instead of being strong upvoted, for a couple of reasons.
Sometimes I just want to make a not very important or not very insightful comment, which doesn’t really deserve to have a higher karma (or to attract more attention) than many other comments, but if I manually drop it to normal upvote, people might interpret that as others having downvoted it.
Having a initially high karma on a post or comment makes it less likely for others to upvote it further (or reduces the total amount of upvotes from others), which reduces the positive feedback from making posts and comments for people who have high karma / voting power.
(I think I might actually prefer to go back to the LW1 system we had at the end, but I’m not sure if that’s a rational response or if I’m just used to it and don’t like change anymore.)
(the “initially high karma will make people less likely to upvote things” is an interesting concern that hadn’t occurred to me. I’m not sure how to best test it, but thanks for bringing it up as a thing to watch out for)
Another issue that I don’t know if you’ve thought of: strong votes from people with high karma are not very anonymous. If I’m talking with someone and I strong up/down vote them (like here recently), it’s pretty obvious who did it, yet not fully common knowledge, which makes the situation even more taxing on the social part of my brain than just straightforward non-anonymous voting (like on Facebook).
Yeah, that is something we’ve thought about, and aren’t sure how to think about. We’ve considered straight-up requiring Strong Upvotes to be an actual endorsement that comes with a name attached. (We’ve also considered actually just making all votes public).
Most of the options come with some pros and cons that weigh together in subtle ways.
My subjective experience in the first 2 weeks has generally been to not know who is voting on what – theoretically I could keep track of everyone’s strong-upvote strength. Maybe over time I’d come to know them well enough that I’d start to track that automatically, for now it just feels lossy enough that I don’t notice.
Having a high amount of voting power basically feels like a disadvantage to me instead of a benefit, because it makes me more reluctant to exercise strong voting power. Maybe other people won’t be able to tell who voted on something, but the part of my brain that worries about this kind of thing isn’t really mollified by your data point.
Maybe as a compromise that ameliorates both of the problems I mentioned, consider capping the strong voting power of everyone at some low amount, like 5?
Good to know. (I do want to stay in explore-options and gather data mode rather than leap towards any particular solution but want you know I’m taking this pretty seriously).
I’m curious how you feel about solutions that are more in the direction of “make karma more illegible?” – possibly literally just randomizing +/- a few points?
There’s also the option of “you can upvote UP TO your max-karma, but can choose any amount”. The two reasons we didn’t go with this was that it felt like asking people to making too-granular a choice every time they voted, and didn’t let us move in the direction of eigenkarma. But it’s still one of the more obvious things to try.
I do want to stay in explore-options and gather data mode
What kind of analysis are you thinking of doing on the data that you’re gathering? I’m curious, and also pre-registration may be a good idea in situations like this to reduce bias.
I’m curious how you feel about solutions that are more in the direction of ″make karma more illegible?″ – possibly literally just randomizing +/- a few points?
Doesn’t seem like it helps as much as other possibilities and the cost seems substantial (in implementation, user education, getting used to it psychologically, maybe other unforeseen consequences).
There’s also the option of “you can upvote UP TO your max-karma, but can choose any amount”. The two reasons we didn’t go with this was that it felt like asking people to making too-granular a choice every time they voted
I think I may share some of the intuitions motivating this, although I like to get more data before changing course. (I also have intuitions running in the opposite direction)
I do think this is going to feel unnaturally for awhile no matter what, and want to give it time to feel more-like-normal before drawing major conclusions.
Having a initially high karma on a post or comment makes it less likely for others to upvote it further (or reduces the total amount of upvotes from others)
My interpretation was “if people have a rough notion of how much karma something is ‘supposed’ to have, then if it already starts out at, say, 9, people will be less motivated to upvote it than if it started lower”.
Laziness—the marginal benefit of voting on something decreases with the absolute value of its current karma, but the cost of voting stays constant.
To prevent the “rich get richer” phenomenon, where if everyone pays more attention to posts/comments that have higher karma, but votes independently without regard to current karma, a comment or post that initially gets some upvotes would attract more attention and more upvotes. Similarly, if everyone did that, comments that arrive late or are in deeper threads or in unpopular posts will have much lower karma than more visible comments of similar quality. In other words, the karma would reflect visibility*quality rather than just quality and it would be hard for readers to recover the quality signal from that.
Doesn’t this attitude obviously defeat the entire purpose of the voting system?!
… if everyone pays more attention to posts/comments that have higher karma, but votes independently without regard to current karma …
Do people do this? They shouldn’t! (I certainly don’t.)
(You might say: “yes, Said, maybe you don’t, but most people do, and we’re not talking about your should-world”; but actually, we are talking about exactly that, aren’t we? So the question is whether that should-world is consistent. I think it is.)
Laziness …
That does not seem to have much to do with “how much karma something should have”, though. (Or rather, it does, but only if you already assume a notion of “how much karma something should have”; laziness cannot explain this notion.)
Doesn’t this attitude obviously defeat the entire purpose of the voting system?!
Can you say more about this?
Karma is supposed to aggregate the commentariat’s opinions of posts/comments. If votes don’t express those opinions in the first place, then there’s nothing to aggregate and karma becomes pointless.
My rough guess from user interviews is that between 30% and 60% of users view karma that way (I tend to flip back and forth between the interpretations). I do not think it defeats the purpose of the voting system (currently at a retreat so can’t write a ton on why I think this, but I don’t think it’s obvious that it should)
My main answer is “you get more signal out of the system if people assume votes have a ‘correct’ amount.”
Say we _don’t_ have strong upvote / small upvotes. There’s just a single upvote.
You see two comments: one pretty good, one great.
Say there are 100 clones of yourself using LW.
Would you rather both comments get 100 karma? Or the better comment to end up with more?
The only options here are
all 100 clones upvote both equally
all 100 clones only upvote the ‘great’ comment
all 100 clones run some sort of percentage-chance-to-upvote on the ‘pretty-good’ comment
all 100 clones upvote the pretty-good-comment if it has less than 35 karma.
The 3rd and 4th option seem roughly equivalent to me.
Variable-size-votes do enable the possibility of a finer-grained result with everyone voting purely based on their individual assessments, but if everyone voted for a comment exactly based on how good they thought it was, and you _add_ those votes up, then the karma ends up being more a function of “how many people saw the comment” than “how good it is.”
(Wei_Dai’s comments on ‘cost of voting’ and ‘preventing rich-getting-richer’ also play into it)
If you want an idealized system wherein the expectation is “everyone votes their true appraisal of a comment’s value”, then the way I’d implement it would be:
1. Comment karma is hidden until you vote on it (so you don’t anchor as much)
2. The karma displayed isn’t the total, but something more like an average (or a function that’s closer to an average than a total).
(I think this might actually be a good system, although it’s a pretty big change)
I also think that this is likely to be a pretty good system. (I can the outlines of some downsides, but would have to think further before I could describe them satisfactorily; in any case, it does not seem to me, at first glance, that any downsides of this system would exceed those of the current system. But this is not yet a strongly held opinion.)
Are these the only users, or is it me and my 100 clones and also some other users? I’m going to assume the latter, because in the former case the entire karma system is just pointless; correct me if my assumption is mistaken.
In that case, I strongly prefer the 3rd option. (The 2nd is an acceptable fallback position if the 3rd is unavailable.) The 3rd and 4th options do not seem at all equivalent to me; it seems strange to suggest otherwise.
My assumption here is that the outcome of options #3 and #4 are fairly similar. If Great Comment is 3x as good as Good comment, then option #3 looks something like “33% chance of upvoting” and option #4 is “upvote if it’s less than about 33 karma. Possibly downvote if it’s got so much karma that it’s getting sorted higher than the Great Comment.”
How similar the outcomes are depends on some of the precise numbers, but I think the order-of-magnitude* is about the same.
*where by order-of-magnitude I mean “base 4 order of magnitude”, which is the comparison that usually seems most relevant to me.
Option #4 is not “thread-safe”. It therefore can, and very likely will, cause chaotic behavior with unpredictable and potentially quite undesirable attractors.
Edit: Note that we may already observe this taking place.
Doublechecking what you mean by “thread-safe” (multiple people looking at the same thing at the same time making decisions before they see what other people do?)
Cool. That doesn’t seem like that big an issue to me, because the system has built-in error correction – people come back later, see that it’s slid past whichever direction they thought it should go, and can change their vote. It more robustly converges on Good Comment getting a proportionally correct-ish score (whereas the “roll a die in your head, upvote 33% of the time” will some non-trivial portion of the time result in things getting way-the-hell upvoted (or not) that should have been).
I should note: I don’t all think this is necessarily the best approach, just, it’s an approach that seems “reasonable” enough that describing it as ‘defeating the point of the voting system’ doesn’t seem accurate.
The core of the problem remains: it requires users to know what other users are doing (as well as how many other users there are, and how many other users are paying attention to a comment, and other such things). The cognitive overhead is tremendously higher. The potential for error is (thus) also much higher.
I want to suggest having new posts/comments be normal upvoted by default (by the author) instead of being strong upvoted, for a couple of reasons.
Sometimes I just want to make a not very important or not very insightful comment, which doesn’t really deserve to have a higher karma (or to attract more attention) than many other comments, but if I manually drop it to normal upvote, people might interpret that as others having downvoted it.
Having a initially high karma on a post or comment makes it less likely for others to upvote it further (or reduces the total amount of upvotes from others), which reduces the positive feedback from making posts and comments for people who have high karma / voting power.
(I think I might actually prefer to go back to the LW1 system we had at the end, but I’m not sure if that’s a rational response or if I’m just used to it and don’t like change anymore.)
(the “initially high karma will make people less likely to upvote things” is an interesting concern that hadn’t occurred to me. I’m not sure how to best test it, but thanks for bringing it up as a thing to watch out for)
Another issue that I don’t know if you’ve thought of: strong votes from people with high karma are not very anonymous. If I’m talking with someone and I strong up/down vote them (like here recently), it’s pretty obvious who did it, yet not fully common knowledge, which makes the situation even more taxing on the social part of my brain than just straightforward non-anonymous voting (like on Facebook).
Yeah, that is something we’ve thought about, and aren’t sure how to think about. We’ve considered straight-up requiring Strong Upvotes to be an actual endorsement that comes with a name attached. (We’ve also considered actually just making all votes public).
Most of the options come with some pros and cons that weigh together in subtle ways.
My subjective experience in the first 2 weeks has generally been to not know who is voting on what – theoretically I could keep track of everyone’s strong-upvote strength. Maybe over time I’d come to know them well enough that I’d start to track that automatically, for now it just feels lossy enough that I don’t notice.
Having a high amount of voting power basically feels like a disadvantage to me instead of a benefit, because it makes me more reluctant to exercise strong voting power. Maybe other people won’t be able to tell who voted on something, but the part of my brain that worries about this kind of thing isn’t really mollified by your data point.
Maybe as a compromise that ameliorates both of the problems I mentioned, consider capping the strong voting power of everyone at some low amount, like 5?
Good to know. (I do want to stay in explore-options and gather data mode rather than leap towards any particular solution but want you know I’m taking this pretty seriously).
I’m curious how you feel about solutions that are more in the direction of “make karma more illegible?” – possibly literally just randomizing +/- a few points?
There’s also the option of “you can upvote UP TO your max-karma, but can choose any amount”. The two reasons we didn’t go with this was that it felt like asking people to making too-granular a choice every time they voted, and didn’t let us move in the direction of eigenkarma. But it’s still one of the more obvious things to try.
What kind of analysis are you thinking of doing on the data that you’re gathering? I’m curious, and also pre-registration may be a good idea in situations like this to reduce bias.
Doesn’t seem like it helps as much as other possibilities and the cost seems substantial (in implementation, user education, getting used to it psychologically, maybe other unforeseen consequences).
I agree with this.
I think I may share some of the intuitions motivating this, although I like to get more data before changing course. (I also have intuitions running in the opposite direction)
I do think this is going to feel unnaturally for awhile no matter what, and want to give it time to feel more-like-normal before drawing major conclusions.
Why would this be the case…?
My interpretation was “if people have a rough notion of how much karma something is ‘supposed’ to have, then if it already starts out at, say, 9, people will be less motivated to upvote it than if it started lower”.
… does anyone actually think like this? (Doesn’t this attitude obviously defeat the entire purpose of the voting system?!)
I do this myself for a couple of reasons:
Laziness—the marginal benefit of voting on something decreases with the absolute value of its current karma, but the cost of voting stays constant.
To prevent the “rich get richer” phenomenon, where if everyone pays more attention to posts/comments that have higher karma, but votes independently without regard to current karma, a comment or post that initially gets some upvotes would attract more attention and more upvotes. Similarly, if everyone did that, comments that arrive late or are in deeper threads or in unpopular posts will have much lower karma than more visible comments of similar quality. In other words, the karma would reflect visibility*quality rather than just quality and it would be hard for readers to recover the quality signal from that.
Can you say more about this?
Do people do this? They shouldn’t! (I certainly don’t.)
(You might say: “yes, Said, maybe you don’t, but most people do, and we’re not talking about your should-world”; but actually, we are talking about exactly that, aren’t we? So the question is whether that should-world is consistent. I think it is.)
That does not seem to have much to do with “how much karma something should have”, though. (Or rather, it does, but only if you already assume a notion of “how much karma something should have”; laziness cannot explain this notion.)
Karma is supposed to aggregate the commentariat’s opinions of posts/comments. If votes don’t express those opinions in the first place, then there’s nothing to aggregate and karma becomes pointless.
My rough guess from user interviews is that between 30% and 60% of users view karma that way (I tend to flip back and forth between the interpretations). I do not think it defeats the purpose of the voting system (currently at a retreat so can’t write a ton on why I think this, but I don’t think it’s obvious that it should)
This is shocking to me, and I would absolutely love to hear more about this perspective.
My main answer is “you get more signal out of the system if people assume votes have a ‘correct’ amount.”
Say we _don’t_ have strong upvote / small upvotes. There’s just a single upvote.
You see two comments: one pretty good, one great.
Say there are 100 clones of yourself using LW.
Would you rather both comments get 100 karma? Or the better comment to end up with more?
The only options here are
all 100 clones upvote both equally
all 100 clones only upvote the ‘great’ comment
all 100 clones run some sort of percentage-chance-to-upvote on the ‘pretty-good’ comment
all 100 clones upvote the pretty-good-comment if it has less than 35 karma.
The 3rd and 4th option seem roughly equivalent to me.
Variable-size-votes do enable the possibility of a finer-grained result with everyone voting purely based on their individual assessments, but if everyone voted for a comment exactly based on how good they thought it was, and you _add_ those votes up, then the karma ends up being more a function of “how many people saw the comment” than “how good it is.”
(Wei_Dai’s comments on ‘cost of voting’ and ‘preventing rich-getting-richer’ also play into it)
If you want an idealized system wherein the expectation is “everyone votes their true appraisal of a comment’s value”, then the way I’d implement it would be:
1. Comment karma is hidden until you vote on it (so you don’t anchor as much)
2. The karma displayed isn’t the total, but something more like an average (or a function that’s closer to an average than a total).
(I think this might actually be a good system, although it’s a pretty big change)
I also think that this is likely to be a pretty good system. (I can the outlines of some downsides, but would have to think further before I could describe them satisfactorily; in any case, it does not seem to me, at first glance, that any downsides of this system would exceed those of the current system. But this is not yet a strongly held opinion.)
Are these the only users, or is it me and my 100 clones and also some other users? I’m going to assume the latter, because in the former case the entire karma system is just pointless; correct me if my assumption is mistaken.
In that case, I strongly prefer the 3rd option. (The 2nd is an acceptable fallback position if the 3rd is unavailable.) The 3rd and 4th options do not seem at all equivalent to me; it seems strange to suggest otherwise.
My assumption here is that the outcome of options #3 and #4 are fairly similar. If Great Comment is 3x as good as Good comment, then option #3 looks something like “33% chance of upvoting” and option #4 is “upvote if it’s less than about 33 karma. Possibly downvote if it’s got so much karma that it’s getting sorted higher than the Great Comment.”
How similar the outcomes are depends on some of the precise numbers, but I think the order-of-magnitude* is about the same.
*where by order-of-magnitude I mean “base 4 order of magnitude”, which is the comparison that usually seems most relevant to me.
Option #4 is not “thread-safe”. It therefore can, and very likely will, cause chaotic behavior with unpredictable and potentially quite undesirable attractors.
Edit: Note that we may already observe this taking place.
Doublechecking what you mean by “thread-safe” (multiple people looking at the same thing at the same time making decisions before they see what other people do?)
Right.
Cool. That doesn’t seem like that big an issue to me, because the system has built-in error correction – people come back later, see that it’s slid past whichever direction they thought it should go, and can change their vote. It more robustly converges on Good Comment getting a proportionally correct-ish score (whereas the “roll a die in your head, upvote 33% of the time” will some non-trivial portion of the time result in things getting way-the-hell upvoted (or not) that should have been).
I should note: I don’t all think this is necessarily the best approach, just, it’s an approach that seems “reasonable” enough that describing it as ‘defeating the point of the voting system’ doesn’t seem accurate.
The core of the problem remains: it requires users to know what other users are doing (as well as how many other users there are, and how many other users are paying attention to a comment, and other such things). The cognitive overhead is tremendously higher. The potential for error is (thus) also much higher.