Even if every participant answered all 21 questions, and if every participant answers with respect to some total ordering, then this still only reduces to the problem of preferential voting. This is a really hard problem and I’m not aware of any voting system that’s provably “correct”—in fact, there are results that for some definitions of “correct”, there are no correct voting systems period. There are various voting systems that satisfy some, but not all, reasonable properties you could name, and you should pick one.
“Ranked pairs” is one natural choice (I don’t know how well it performs in practice, though) because it deals in pairs to begin with. Basically, for each pair (A,B) you tally how many participants picked A over B. You go through the pairs in order of how significant a majority they represent. In this case, you wouldn’t want to use the usual method for deciding which majority is better, because you have to take into account the number of people who answered the question, too. There was a post on LW once which linked to an article about this (in the context of e.g. ratings of movies on IMDB, where a movie rated 10.0 by 5 users isn’t as good as a movie rated 9.0 by 100 users). Anyway, the next thing you do with the pairs is lock them in one by one, except when it would create a cycle together with pairs that have already been locked in. If you do this, the “locked in” pairs will show the most preferred object.
Upvoted. This problem is the well-studied voting problem.
there are results that for some definitions of “correct”, there are no correct voting systems period.
In particular, the seminal result is Arrow’s impossibility theorem, which states that no voting system satisfies all of the following criteria: (from Wikipedia)
If every voter prefers alternative X over alternative Y, then the group prefers X over Y.
If every voter’s preference between X and Y remains unchanged, then the group’s preference between X and Y will also remain unchanged.
There is no “dictator”: no single voter possesses the power to always determine the group’s preference.
Even if every participant answered all 21 questions, and if every participant answers with respect to some total ordering, then this still only reduces to the problem of preferential voting. This is a really hard problem and I’m not aware of any voting system that’s provably “correct”—in fact, there are results that for some definitions of “correct”, there are no correct voting systems period. There are various voting systems that satisfy some, but not all, reasonable properties you could name, and you should pick one.
“Ranked pairs” is one natural choice (I don’t know how well it performs in practice, though) because it deals in pairs to begin with. Basically, for each pair (A,B) you tally how many participants picked A over B. You go through the pairs in order of how significant a majority they represent. In this case, you wouldn’t want to use the usual method for deciding which majority is better, because you have to take into account the number of people who answered the question, too. There was a post on LW once which linked to an article about this (in the context of e.g. ratings of movies on IMDB, where a movie rated 10.0 by 5 users isn’t as good as a movie rated 9.0 by 100 users). Anyway, the next thing you do with the pairs is lock them in one by one, except when it would create a cycle together with pairs that have already been locked in. If you do this, the “locked in” pairs will show the most preferred object.
Upvoted. This problem is the well-studied voting problem.
In particular, the seminal result is Arrow’s impossibility theorem, which states that no voting system satisfies all of the following criteria: (from Wikipedia)
Thanks—the voting system analogy didn’t occur to me. Reading up on ranked pairs: http://en.wikipedia.org/wiki/Ranked_pairs