Good point about inconsistency… I was thinking that individual responses may be inconsistent, but the aggregated responses of the group might reveal a significant preference.
My first crack at this was to use a simple voting system, where B from {A,B} means +1 votes for B, 0 for A, largest score when all participant votes are tallied wins. What messes that up is participants leaving without completing the entire set, which introduces selection bias, even if the sets are served at random.
Preference ordering / ranking isn’t ideal because it takes longer, which may encourage the participant to rationalize. The “pick-one” approach is designed to be fast, which is better for gut reactions when comparing words, photos, etc.
If the aggregated preferences are transitive (i.e., ‘not inconsistent’ in your and Manfred’s wording), then this preference relation defines a total order on the objects, and there is a unique object that is preferred to every other object (in aggregate). (Furthermore, this is isomorphic to the set {1,2,3,...,7} under the ≤ relation.)
As I understand things, you have no guarantee of transitivity in the aggregated preferences even if you do have transitivity in the individual preferences.
Good point about inconsistency… I was thinking that individual responses may be inconsistent, but the aggregated responses of the group might reveal a significant preference.
My first crack at this was to use a simple voting system, where B from {A,B} means +1 votes for B, 0 for A, largest score when all participant votes are tallied wins. What messes that up is participants leaving without completing the entire set, which introduces selection bias, even if the sets are served at random.
Preference ordering / ranking isn’t ideal because it takes longer, which may encourage the participant to rationalize. The “pick-one” approach is designed to be fast, which is better for gut reactions when comparing words, photos, etc.
If the aggregated preferences are transitive (i.e., ‘not inconsistent’ in your and Manfred’s wording), then this preference relation defines a total order on the objects, and there is a unique object that is preferred to every other object (in aggregate). (Furthermore, this is isomorphic to the set {1,2,3,...,7} under the ≤ relation.)
As I understand things, you have no guarantee of transitivity in the aggregated preferences even if you do have transitivity in the individual preferences.
Yes, of course you are correct.
Very helpful—reading about this now. Starting here: http://en.wikipedia.org/wiki/Ranking