The posts available for review are presented in (what I guess is) a consistent order that is (so far as I know) the same for everyone. I expect this to mean that posts presented earlier will get more votes. If, as seems plausible, most of the things in the list aren’t bad enough to get a “no” vote from most voters, this means that there is a bias in favour of the earlier posts in the list.
Related: keeping track of which posts I’ve looked at is a bit of a pain. Obviously I can see which ones I’ve voted non-neutral for, but there’s no obvious way to distinguish “decided to stick with neutral” from “haven’t looked yet”. So long as the order of presentation is consistent, I can just remember how far through the list I am, but (see above) it’s not obviously a good thing for the order of presentation to be consistent. And this phenomenon incentivizes me to process posts in order, rather than deliberately counteracting the bias mentioned above by trying to look at them in a random order.
The list is now shuffled (as a tiebreak after sorting by your own vote). The shuffle is done once per user, so each user should see the posts in a random order, but it’ll be the same order each time you revisit it. This change went live around the 13th.
[EDIT: When I say that posts earlier in the list got 25-50% more votes, I mean simply the number of non-neutral votes cast on those items, regardless of direction of magnitude. It would perhaps be more accurate to say these posts had 25-50% more people vote on them.]
The posts available for review are presented in (what I guess is) a consistent order that is (so far as I know) the same for everyone. I expect this to mean that posts presented earlier will get more votes.
Good call. I looked into this and found an effect of somewhere between 25-50% more votes for posts being displayed earlier in the list. The team rolled out a fix to randomize loading this morning.
Interestingly, the default sort order was by number of nominations in ascending, so the most heavily nominated (approx, the most popular) posts were being displayed last. These posts were getting as many votes as those at the beginning of the list (though possibly not as many as they might have otherwise), and it’s the posts in the middle that were getting less.
This was an oversight which we’re glad to have caught. We’ve around halfway through the voting, plus the second half will have the deadline rush, so hopefully this bias will get countered in the coming week.
The posts available for review are presented in (what I guess is) a consistent order that is (so far as I know) the same for everyone. I expect this to mean that posts presented earlier will get more votes. If, as seems plausible, most of the things in the list aren’t bad enough to get a “no” vote from most voters, this means that there is a bias in favour of the earlier posts in the list.
Related: keeping track of which posts I’ve looked at is a bit of a pain. Obviously I can see which ones I’ve voted non-neutral for, but there’s no obvious way to distinguish “decided to stick with neutral” from “haven’t looked yet”. So long as the order of presentation is consistent, I can just remember how far through the list I am, but (see above) it’s not obviously a good thing for the order of presentation to be consistent. And this phenomenon incentivizes me to process posts in order, rather than deliberately counteracting the bias mentioned above by trying to look at them in a random order.
The list is now shuffled (as a tiebreak after sorting by your own vote). The shuffle is done once per user, so each user should see the posts in a random order, but it’ll be the same order each time you revisit it. This change went live around the 13th.
[EDIT: When I say that posts earlier in the list got 25-50% more votes, I mean simply the number of non-neutral votes cast on those items, regardless of direction of magnitude. It would perhaps be more accurate to say these posts had 25-50% more people vote on them.]
Good call. I looked into this and found an effect of somewhere between 25-50% more votes for posts being displayed earlier in the list. The team rolled out a fix to randomize loading this morning.
Interestingly, the default sort order was by number of nominations in ascending, so the most heavily nominated (approx, the most popular) posts were being displayed last. These posts were getting as many votes as those at the beginning of the list (though possibly not as many as they might have otherwise), and it’s the posts in the middle that were getting less.
This was an oversight which we’re glad to have caught. We’ve around halfway through the voting, plus the second half will have the deadline rush, so hopefully this bias will get countered in the coming week.
Unfortunately you just make mistakes the first time you’re doing things. :/