One of the arguments is quite misleading in most cases, so probably not high-quality by typical definitions. Unfortunately, under the time limit, our readers can’t reliably tell which one is misleading.
I don’t know about anyone else; under time pressure I personally would go about looking for ‘one wrong argument in a sea of high-quality arguments’ very differently than I would go about looking for ‘one misleading but superficially high-quality argument in a sea of high-quality arguments’ or ‘one very-high-quality argument in a sea of high-quality arguments’.
One of the arguments is quite misleading in most cases, so probably not high-quality by typical definitions. Unfortunately, under the time limit, our readers can’t reliably tell which one is misleading.
Without arguments and without the time limit, annotators get the questions right with ~90% accuracy: https://arxiv.org/abs/2112.08608
Did your description to the participants state that the arguments were high-quality?
I can look up the exact wording if it’s helpful, but I assume it’s clear from the basic setup that at least one of the arguments has to be misleading.
I don’t know about anyone else; under time pressure I personally would go about looking for ‘one wrong argument in a sea of high-quality arguments’ very differently than I would go about looking for ‘one misleading but superficially high-quality argument in a sea of high-quality arguments’ or ‘one very-high-quality argument in a sea of high-quality arguments’.