You can learn more details about the structure of the challenge in our paper.
I had a look:
For inputs upon which the model does not abstain, the model must never make a mistake.
More on that:
To break a defense, an attacker must
submit an unambiguous image of either a bird or bicycle that the model labels incorrectly (as decided by an ensemble of human judges). Models are allowed to abstain (by returning low-confidence predictions on any adversarial input6, but even one confident error will result in a broken defense.
That note 6:
Models are prevented from always abstaining through the abstaining mechanism described in Section ??
Which seems to be here:
4.2 Abstaining mechanism for the contest
In the contest, defenders attempt to create a model that never makes a confident mistake. Although the task is binary (A vs. B), models are allowed to output three labels: “Class A”, “Class B”, or “Abstain”. We define a confident mistake as the model assigning one class label, when a unanimous ensemble of human taskers give the other class label. Abstaining on all adversarial inputs is acceptable, and we allow defenses to choose when to abstain using any mechanism they desire. To prevent models from abstaining on all inputs, models must reach 80% accuracy on a private eligibility dataset of clean bird-or-bicycle images. We reserve the right to perform additional tests to ensure defenders do not over-fit against our private held-out data [Markoff, 2015].
Briefly: Yes, but the ‘eligibility data set’ is private, and they can test a winner more at their discretion*.
*The exact line is at the end of this post, in bold.
Long: (What they say, and how to find it.)
From the website:
I had a look:
More on that:
That note 6:
Which seems to be here:
Emphasis Mine.