I’d like to draw people’s attention to a couple of recent “karma anomalies”. I think these show a worrying tendency for arguments that support the majority LW opinion to accumulate karma regardless of their actual merits.
Exhibit A. I gave a counterargument which convinced the author of that comment to change his mind, yet the original comment is still at 14.
Exhibit B. James Andrix’s comment is at 20, while Toby Ord’s counterargument is at 3. This issue is still confusing to me so I can’t say for sure that Toby is right and James is wrong, but I think Toby has the stronger argument, and in any case I see no way that 20 to 3 is justified on the merits.
ETA: Please do not vote down these comments due to this discussion. My intention is to find a fix for a systemic problem, not to cause these particular comments to be voted down.
I’ve noticed in general that later replies to comments get less votes; it’s possibly because fewer people are still reading. Support for this is that your comment on the other thread already has 7 points, and all of this should go to Toby Ord.
Also, in Exhibit B, James Andrix gave an example attacking religion, which likely got him some votes (for attacking the Great Enemy), and since Toby Ord didn’t support his argument, this probably stopped him from getting votes, since by so doing, he defended the Enemy, which is treason.
Wedrifid’s comment, “He did not get my vote because his final paragraph about the bible/koran is distinctly muddled thinking,” provides more support for the point about Exhibit B.
Eh, that’s not a very generous reading of what he wrote. Exhibit A has a post at very high karma despite arguments that convinced its own author to drop support for it. That’s not karma “working,” either.
For some implicit definition of karma ‘working’ that is unclear. Absent a bug in the karma scoring code, a discrepancy between the karma scores you observe and the karma scores you think are warranted seems just as likely to be an inaccuracy in the observer’s model of how karma is supposed to work as a problem with the karma system.
What the original post seems to be missing to me is an explanation of what scores the karma system should be producing for these posts, a justification for why that is what the karma system should be producing and ideally a suggestion for changes to either the implementation of the system or the way people allocate their votes that would produce the desired changes. Absent the above it look a lot like complaining that people aren’t voting the way you think they ought to.
Well, to start with I wanted to see if others agree that a problem exists here. If most people are satisfied with how karma is working in these cases, then there is not much point in me spending a lot of time writing out long explanations and justifications, and trying to find solutions. So at this stage, I’m basically saying “This looks wrong to me. What do you think?” I think I did give some explanations and justifications, but I accept that more are needed if an eventual change to the karma system is to be made.
Ok, as one data point, I don’t see a particular problem here. The higher rated posts in your examples deserved higher ratings in my opinion. Karma mostly functions as I expect it to function.
Thanks, but can you explain why you think people who post wrong arguments deserve to get more karma than those who correct the wrong arguments? Suppose I thought of uninverted’s argument, but then realized that it’s wrong, so I don’t post my original argument, and instead correct him when he posts his. I end up with less karma than if I hadn’t spent time thinking things through and realizing the flaw in my reasoning. Why do we want to discourage “less wrong” thinking in this way?
It seems to me that the way karma works now encourages people to think up arguments that support the majority view and then post them as soon as they can without thinking things through. Why is this good, or “expected”?
First, I think you’re missing a karma pattern that I’ve noticed which is that the first post in a thread often gets more extreme votes (scores of greater absolute magnitude) than subsequent posts. I imagine this is because more people read the earlier posts in a thread and interest/readership drops off the deeper the nesting gets. I don’t see any simple way to ‘fix’ that—it has the potential to be gamed but I don’t think gaming the system in that respect is a major problem here.
Second I don’t think karma strictly reflects ‘correctness’ of arguments, nor do I think it necessarily should. People award karma for attributes other than correctness. For example I imagine some of the upvotes on uninverted’s “But I don’t want to be a really big integer!” comment were drive-by upvotes for an amusing remark. Some of those upvoters won’t have stayed for the followup discussion, others might have awarded more karma for pithy and amusing than accurate but dry. I think points-for-humour is as likely an explanation here as points-for-majority-opinion. Maybe you don’t think karma should be awarded for attributes other than correctness. If so, go ahead and bring it up and see what the rest of the community thinks.
As a side note, I think you probably shouldn’t have chosen a thread where you were a participant as an example. It gives the slight impression that your real complaint is that uninverted got more brownie points than you even though you were right and it’s just not fair. If I didn’t recognize your username as a regular and generally high-value contributor I might not have given you the benefit of the doubt on that.
As a side note, I think you probably shouldn’t have chosen a thread where you were a participant as an example. It gives the slight impression that your real complaint is that uninverted got more brownie points than you even though you were right and it’s just not fair.
I was given that impression somewhat but then on reflection I realized a more likely prompt for the Wei’s frustration was the Nelson/Komponisto/Knox affair. Not wanting to bring that issue up yet again, he chose some other similar examples that didn’t come with as much baggage. That one of them was his own was just unfortunate.
That would presumably require a fairly different rating system, under the current system median karma would mean posts could only ever score −1, 0 or 1. That doesn’t seem like an improvement to me.
the first post in a thread often gets more extreme votes (scores of greater absolute magnitude) than subsequent posts.
In that the main effect of karma on the reader is to sort posts, comparing scores at different levels of nesting is irrelevant. It is a very biased heuristic to read only comments at, say, karma > 5. I don’t know if anyone uses this heuristic. A lot of people only read comments at karma > −4, which probably has a similar bias, but I wouldn’t worry about it.
because more people read the earlier posts in a thread and interest/readership drops off the deeper the nesting gets.
People who post replies are serving a smaller audience than people who post higher level comments. It is possibly good that they are proportionately rewarded with karma.
I don’t see any simple way to ‘fix’ that—it has the potential to be gamed but I don’t think gaming the system in that respect is a major problem here
It’s not so much a potential to be gamed, as encouraging people to post without thinking things through, as well as misleading readers as to which arguments are correct. I don’t know if there is a simple fix or not, but if we can agree that it’s a problem, then we can at least start thinking about possible solutions.
Maybe you don’t think karma should be awarded for attributes other than correctness. If so, go ahead and bring it up and see what the rest of the community thinks.
In a case where a comment is both funny and incorrect, I think we should prioritize the correctness. After all, this is “Less Wrong”, not “Less Bored”.
If I didn’t recognize your username as a regular and generally high-value contributor I might not have given you the benefit of the doubt on that.
I was too lazy to find another example, and counting on the benefit of the doubt. :)
ETA: Also, I think being upvoted for supporting the majority opinion is clearly a strong reason for what happened, especially in Exhibit B, where the comment is deep in the middle of a thread, and has no humor value.
It’s not so much a potential to be gamed, as encouraging people to post without thinking things through, as well as misleading readers as to which arguments are correct.
I’m not sure that’s true. As I originally said, the first comment in a thread often gets karma of greater absolute magnitude. Bad posts get voted down more harshly as well as good posts getting voted up more. I think the higher readership for top level comments explains this. It means that from a karma-gaming perspective posting a top level comment is only a good move if you are confident it will be received positively.
In a case where a comment is both funny and incorrect, I think we should prioritize the correctness.
How about other attributes not directly related to correctness? How should niceness be judged relative to correctness for example?
I still think there’s a conflict between you wanting people to give more upvotes to things that are correct but fewer to things that ‘agree with the majority opinion’. I don’t think people upvote because a comment ‘agrees with the majority opinion’, they upvote because a comment agrees with their opinion. That tends to produce greater upvotes for the majority view. In your second example I think the greater upvotes for James Andrix reflect the fact that he is more correct. Your real complaint seems to be that the majority opinion is wrong on this issue. The best way to fix that is to make a better argument for the other view, not to complain that people are failing to recognize correct arguments and upvote them.
Ok, as one data point, I don’t see a particular problem here. The higher rated posts in your examples deserved higher ratings in my opinion. Karma mostly functions as I expect it to function.
Thanks, but can you explain why you think people who post wrong arguments deserve to get more karma than those who correct the wrong arguments?
Mattnewport did not claim or otherwise imply that he thought that.
I agree with you, but I think it has to do with the way people vote (mainly voting in favor of things they agree with and against things they disagree with), and with which comments are read by whom. In other words, changing the karma system probably is not a way to address it: people have to change their behavior.
It seems a little inconsistent to expect people to vote up things for being correct but not to vote up things simply because they agree with them. I tend to agree with things I think are correct and disagree with things I think are incorrect.
I think Wei Dai was saying that people should vote up strong arguments, even if they disagree with the conclusion. I do this sometimes, and I think it’s a good thing to do.
If you look a little closer you see that ‘the own author’ was persuaded to concede that later comment in the argument and was then more generous and conciliatory than he perhaps needed to be. I would be extremely disappointed if the meta discussion here actually made the author retract his comment. What we have here is a demonstration of why it is usually status-enhancing to treat arguments as soldiers. If you don’t, you’re just giving the ‘enemy’ ammunition.
Willingness to concede weak points in a position is a rare trait and one that I like to encourage. This means I will never use ‘look, he admitted he was wrong’ as way to coerce people into down-voting them or shame those that don’t.
EDIT: I mean status enhancing specifically not rational in general.
Willingness to concede weak points in a position is a rare trait and one that I like to encourage. This means I will never use ‘look, he admitted he was wrong’ as way to coerce people into down-voting them or shame those that don’t.
That’s a very good point, and I’ve added a note to my opening comment to convey that I don’t want people to down-vote these particular comments.
I think I should point out a problem with the karma system when I see it, and use evidence and arguments to back up my position and gather support and ideas for fixing the problem. I believe that’s how a “community” works.
Nevermind A and B. I’m waiting to see how Exhibit C fares. 11 − 3 at the moment. 12 −15 is what I would expect if each comment received equal exposure. Perhaps discounting the reply a little because interest in the thread may have waned slightly.
Exhibit A. I gave a counterargument which convinced the author of that comment to change his mind, yet the original comment is still at 14.
Exhibit A has my vote because it is a reasonably insightful one liner, and a suitable response to the parent. Your reply to Exhibit A is a reducto ad absurdium that just does not follow.
I pointed out that accepting this premise would lead to indifference between wireheading and anti-wireheading.
Which is simply wrong. Please see this list of preferences which seem natural regarding positive and negative integers (and their wireheading counterparts). You haven’t even expressed disagreement with any of those propositions that I expected to uncontroversial yet your whole ‘karma anomalies’ objection seems to hinge on it. I find this extremely rude.
Exhibit B. James Andrix’s comment is at 20, while Toby Ord’s counterargument is at 3. This issue is still confusing to me so I can’t say for sure that Toby is right and James is wrong, but I think Toby has the stronger argument, and in any case I see no way that 20 to 3 is justified on the merits.
This is an excellent example of the karma system serving its purpose. James’ post was voted up above 20 because it was fascinating. Toby got 5 votes for pointing out the limit to when that kind of math is applicable. He did not get my vote because his final paragraph about the bible/koran is distinctly muddled thinking.
Yes, I was deliberately ignoring you because you were assuming agreement on things that I had already made clear that I don’t agree with. It seemed to me that the discussion wasn’t being productive because you weren’t paying attention. If there is a way to set a disagreement status I would have done that, but apparently the accepted/expected way to end arguments here is to just stop talking.
Also, given your stated taste for “direct social competition”, I’ve decided to not argue with you anymore in the future, since it doesn’t seem to further my ends. Feel free to continue to reply to my comments or posts. I think you often make good points, but debating with you is just not fun. (I reserve the right to agree with you though. :)
Yes, I was deliberately ignoring you because you were assuming agreement on things that I had already made clear that I don’t agree with.
I perhaps should have made myself more clear. One of the points should have been “You assume for the purposes of attempting an argument to absurdity and I similarly assume for the purposes of following through just what the implications are”. It is the intuitive preferences over the states of the universe that I assumed would be shared by most. I also believed that they served to illustrate the bulk of your point.
Also, given your stated taste for “direct social competition”
That was probably an effective move. It seems like honesty is rarely a good policy. Things can always be taken out of context and used against you. In that case of trolls who are being belligerent and silly I do not mind mocking them. I consider such status games completely distinct from discussion. In fact, they are much more like playing lazer tag, another healthy place to turn off the brain and exercise competitive instincts. It is when such status games infect what is presented as ‘intelligent discussion and debate’ that I despise them, usually vocally. It frustrates me furiously when competition degrades conversation to logically rude debate.
I shall continue to agree with you when you make good points, disagree when you make meta level discussions like this one I shall continue to strongly object.
I’d like to draw people’s attention to a couple of recent “karma anomalies”. I think these show a worrying tendency for arguments that support the majority LW opinion to accumulate karma regardless of their actual merits.
Exhibit A. I gave a counterargument which convinced the author of that comment to change his mind, yet the original comment is still at 14.
Exhibit B. James Andrix’s comment is at 20, while Toby Ord’s counterargument is at 3. This issue is still confusing to me so I can’t say for sure that Toby is right and James is wrong, but I think Toby has the stronger argument, and in any case I see no way that 20 to 3 is justified on the merits.
ETA: Please do not vote down these comments due to this discussion. My intention is to find a fix for a systemic problem, not to cause these particular comments to be voted down.
I’ve noticed in general that later replies to comments get less votes; it’s possibly because fewer people are still reading. Support for this is that your comment on the other thread already has 7 points, and all of this should go to Toby Ord.
Also, in Exhibit B, James Andrix gave an example attacking religion, which likely got him some votes (for attacking the Great Enemy), and since Toby Ord didn’t support his argument, this probably stopped him from getting votes, since by so doing, he defended the Enemy, which is treason.
Wedrifid’s comment, “He did not get my vote because his final paragraph about the bible/koran is distinctly muddled thinking,” provides more support for the point about Exhibit B.
By ‘anomaly’ you appear to mean ‘not the scores I would have assigned’. That’s not the way karma works.
Eh, that’s not a very generous reading of what he wrote. Exhibit A has a post at very high karma despite arguments that convinced its own author to drop support for it. That’s not karma “working,” either.
For some implicit definition of karma ‘working’ that is unclear. Absent a bug in the karma scoring code, a discrepancy between the karma scores you observe and the karma scores you think are warranted seems just as likely to be an inaccuracy in the observer’s model of how karma is supposed to work as a problem with the karma system.
What the original post seems to be missing to me is an explanation of what scores the karma system should be producing for these posts, a justification for why that is what the karma system should be producing and ideally a suggestion for changes to either the implementation of the system or the way people allocate their votes that would produce the desired changes. Absent the above it look a lot like complaining that people aren’t voting the way you think they ought to.
Well, to start with I wanted to see if others agree that a problem exists here. If most people are satisfied with how karma is working in these cases, then there is not much point in me spending a lot of time writing out long explanations and justifications, and trying to find solutions. So at this stage, I’m basically saying “This looks wrong to me. What do you think?” I think I did give some explanations and justifications, but I accept that more are needed if an eventual change to the karma system is to be made.
Ok, as one data point, I don’t see a particular problem here. The higher rated posts in your examples deserved higher ratings in my opinion. Karma mostly functions as I expect it to function.
Thanks, but can you explain why you think people who post wrong arguments deserve to get more karma than those who correct the wrong arguments? Suppose I thought of uninverted’s argument, but then realized that it’s wrong, so I don’t post my original argument, and instead correct him when he posts his. I end up with less karma than if I hadn’t spent time thinking things through and realizing the flaw in my reasoning. Why do we want to discourage “less wrong” thinking in this way?
It seems to me that the way karma works now encourages people to think up arguments that support the majority view and then post them as soon as they can without thinking things through. Why is this good, or “expected”?
First, I think you’re missing a karma pattern that I’ve noticed which is that the first post in a thread often gets more extreme votes (scores of greater absolute magnitude) than subsequent posts. I imagine this is because more people read the earlier posts in a thread and interest/readership drops off the deeper the nesting gets. I don’t see any simple way to ‘fix’ that—it has the potential to be gamed but I don’t think gaming the system in that respect is a major problem here.
Second I don’t think karma strictly reflects ‘correctness’ of arguments, nor do I think it necessarily should. People award karma for attributes other than correctness. For example I imagine some of the upvotes on uninverted’s “But I don’t want to be a really big integer!” comment were drive-by upvotes for an amusing remark. Some of those upvoters won’t have stayed for the followup discussion, others might have awarded more karma for pithy and amusing than accurate but dry. I think points-for-humour is as likely an explanation here as points-for-majority-opinion. Maybe you don’t think karma should be awarded for attributes other than correctness. If so, go ahead and bring it up and see what the rest of the community thinks.
As a side note, I think you probably shouldn’t have chosen a thread where you were a participant as an example. It gives the slight impression that your real complaint is that uninverted got more brownie points than you even though you were right and it’s just not fair. If I didn’t recognize your username as a regular and generally high-value contributor I might not have given you the benefit of the doubt on that.
I was given that impression somewhat but then on reflection I realized a more likely prompt for the Wei’s frustration was the Nelson/Komponisto/Knox affair. Not wanting to bring that issue up yet again, he chose some other similar examples that didn’t come with as much baggage. That one of them was his own was just unfortunate.
Upvoted for the correct inference. This is definitely one of those rare times when laziness failed to pay off. :)
Median karma would de-emphasize number of voters and put greater emphasis on the score they assigned.
That would presumably require a fairly different rating system, under the current system median karma would mean posts could only ever score −1, 0 or 1. That doesn’t seem like an improvement to me.
Yes; I imagine a range from 0⁄5 to 5⁄5 as per eg Amazon book rating sites. One problem with this however is that people don’t use the whole range.
In that the main effect of karma on the reader is to sort posts, comparing scores at different levels of nesting is irrelevant. It is a very biased heuristic to read only comments at, say, karma > 5. I don’t know if anyone uses this heuristic. A lot of people only read comments at karma > −4, which probably has a similar bias, but I wouldn’t worry about it.
People who post replies are serving a smaller audience than people who post higher level comments. It is possibly good that they are proportionately rewarded with karma.
It’s not so much a potential to be gamed, as encouraging people to post without thinking things through, as well as misleading readers as to which arguments are correct. I don’t know if there is a simple fix or not, but if we can agree that it’s a problem, then we can at least start thinking about possible solutions.
In a case where a comment is both funny and incorrect, I think we should prioritize the correctness. After all, this is “Less Wrong”, not “Less Bored”.
I was too lazy to find another example, and counting on the benefit of the doubt. :)
ETA: Also, I think being upvoted for supporting the majority opinion is clearly a strong reason for what happened, especially in Exhibit B, where the comment is deep in the middle of a thread, and has no humor value.
I’m not sure that’s true. As I originally said, the first comment in a thread often gets karma of greater absolute magnitude. Bad posts get voted down more harshly as well as good posts getting voted up more. I think the higher readership for top level comments explains this. It means that from a karma-gaming perspective posting a top level comment is only a good move if you are confident it will be received positively.
How about other attributes not directly related to correctness? How should niceness be judged relative to correctness for example?
I still think there’s a conflict between you wanting people to give more upvotes to things that are correct but fewer to things that ‘agree with the majority opinion’. I don’t think people upvote because a comment ‘agrees with the majority opinion’, they upvote because a comment agrees with their opinion. That tends to produce greater upvotes for the majority view. In your second example I think the greater upvotes for James Andrix reflect the fact that he is more correct. Your real complaint seems to be that the majority opinion is wrong on this issue. The best way to fix that is to make a better argument for the other view, not to complain that people are failing to recognize correct arguments and upvote them.
Mattnewport did not claim or otherwise imply that he thought that.
I agree with you, but I think it has to do with the way people vote (mainly voting in favor of things they agree with and against things they disagree with), and with which comments are read by whom. In other words, changing the karma system probably is not a way to address it: people have to change their behavior.
Yes, I agree, and by “karma system” I meant to include how people think they should vote.
It seems a little inconsistent to expect people to vote up things for being correct but not to vote up things simply because they agree with them. I tend to agree with things I think are correct and disagree with things I think are incorrect.
I think Wei Dai was saying that people should vote up strong arguments, even if they disagree with the conclusion. I do this sometimes, and I think it’s a good thing to do.
If you look a little closer you see that ‘the own author’ was persuaded to concede that later comment in the argument and was then more generous and conciliatory than he perhaps needed to be. I would be extremely disappointed if the meta discussion here actually made the author retract his comment. What we have here is a demonstration of why it is usually status-enhancing to treat arguments as soldiers. If you don’t, you’re just giving the ‘enemy’ ammunition.
Willingness to concede weak points in a position is a rare trait and one that I like to encourage. This means I will never use ‘look, he admitted he was wrong’ as way to coerce people into down-voting them or shame those that don’t.
EDIT: I mean status enhancing specifically not rational in general.
That’s a very good point, and I’ve added a note to my opening comment to convey that I don’t want people to down-vote these particular comments.
I think I should point out a problem with the karma system when I see it, and use evidence and arguments to back up my position and gather support and ideas for fixing the problem. I believe that’s how a “community” works.
Sure, and that’s what I felt was missing in the original post.
I’m not sure I see Toby’s argument. James’s I follow.
Nevermind A and B. I’m waiting to see how Exhibit C fares. 11 − 3 at the moment. 12 −15 is what I would expect if each comment received equal exposure. Perhaps discounting the reply a little because interest in the thread may have waned slightly.
Exhibit A has my vote because it is a reasonably insightful one liner, and a suitable response to the parent. Your reply to Exhibit A is a reducto ad absurdium that just does not follow.
Which is simply wrong. Please see this list of preferences which seem natural regarding positive and negative integers (and their wireheading counterparts). You haven’t even expressed disagreement with any of those propositions that I expected to uncontroversial yet your whole ‘karma anomalies’ objection seems to hinge on it. I find this extremely rude.
This is an excellent example of the karma system serving its purpose. James’ post was voted up above 20 because it was fascinating. Toby got 5 votes for pointing out the limit to when that kind of math is applicable. He did not get my vote because his final paragraph about the bible/koran is distinctly muddled thinking.
I wonder if it wouldn’t be more accurate to say that, actually, 98% confidence has been refuted at General Relativity.
Yes, I was deliberately ignoring you because you were assuming agreement on things that I had already made clear that I don’t agree with. It seemed to me that the discussion wasn’t being productive because you weren’t paying attention. If there is a way to set a disagreement status I would have done that, but apparently the accepted/expected way to end arguments here is to just stop talking.
Also, given your stated taste for “direct social competition”, I’ve decided to not argue with you anymore in the future, since it doesn’t seem to further my ends. Feel free to continue to reply to my comments or posts. I think you often make good points, but debating with you is just not fun. (I reserve the right to agree with you though. :)
I perhaps should have made myself more clear. One of the points should have been “You assume for the purposes of attempting an argument to absurdity and I similarly assume for the purposes of following through just what the implications are”. It is the intuitive preferences over the states of the universe that I assumed would be shared by most. I also believed that they served to illustrate the bulk of your point.
That was probably an effective move. It seems like honesty is rarely a good policy. Things can always be taken out of context and used against you. In that case of trolls who are being belligerent and silly I do not mind mocking them. I consider such status games completely distinct from discussion. In fact, they are much more like playing lazer tag, another healthy place to turn off the brain and exercise competitive instincts. It is when such status games infect what is presented as ‘intelligent discussion and debate’ that I despise them, usually vocally. It frustrates me furiously when competition degrades conversation to logically rude debate.
I shall continue to agree with you when you make good points, disagree when you make meta level discussions like this one I shall continue to strongly object.