When I think of a limited judge trying to resolve a debate about a factual question, one of the key behaviors I would check for would be the judge checking on the factual accuracy of the premises of the debaters. As in, looking to make sure the sources that the debaters cite exist, seem credible, and actually seem to agree with the point that the debater is making.
And the debate should be calibrated such that the judge isn’t fully able to make the determination themselves, even with all the sources available.
I recognize that this is out of scope of your work, but I just want to point out what I would see as necessary components to a study which convinced me that debate was working as a method for a ‘weak supervising strong’ approach.
If the correct-side debator uses invalid claims as part of its arguments, and the judge fails to catch this… It would make me feel that something was amiss. That perhaps this wasn’t a good proxy for a high-stakes debator between competent debtors trying to convince a smart and motivated human judge about facts about the world.
And if, given the full set of cited sources from both sides of the debate, the judge is able to consistently come to the correct answer, then the question isn’t hard enough.
When I think of a limited judge trying to resolve a debate about a factual question, one of the key behaviors I would check for would be the judge checking on the factual accuracy of the premises of the debaters. As in, looking to make sure the sources that the debaters cite exist, seem credible, and actually seem to agree with the point that the debater is making.
And the debate should be calibrated such that the judge isn’t fully able to make the determination themselves, even with all the sources available.
I recognize that this is out of scope of your work, but I just want to point out what I would see as necessary components to a study which convinced me that debate was working as a method for a ‘weak supervising strong’ approach.
I agree that this would be a more interesting setup. But why do you see it as necessary to validate the ‘weak supervising strong’ hypothesis?
If the correct-side debator uses invalid claims as part of its arguments, and the judge fails to catch this… It would make me feel that something was amiss. That perhaps this wasn’t a good proxy for a high-stakes debator between competent debtors trying to convince a smart and motivated human judge about facts about the world.
And if, given the full set of cited sources from both sides of the debate, the judge is able to consistently come to the correct answer, then the question isn’t hard enough.