I agree that this would be a more interesting setup. But why do you see it as necessary to validate the ‘weak supervising strong’ hypothesis?
Charlie George
Karma: 23
[Apologies for the really late response]
Testing this trained judge with different debaters, you find that Elo of the debater models and accuracy of the debate result track well with each other. Strangely though, Best-of-4 decoding on the debaters does not seem to increase Elo?
This is strange but the difference in Elo is actually not significant looking at the confidence intervals.
I worry about how much of what we’re seeing is just an effect of domain shift. Since you trained the model on GPT-4o debates, I would expect the accuracy on these debates to be highest, and changing to GPT-4o mini and then GPT-3.5 should lead us further out of domain, reducing the judging model’s accuracy. Then the accuracy trend just reflects how OOD the debates are, and that happens to track with model skill for the debaters you tested. The fact that Elo also tracks in the expected way is a bit harder to explain away here, and makes it seem like the judge is learning something meaningful, but I am pretty unsure about that.
The Elo of the debates stays roughly the same with an untrained judge. So another way you could plot this graph is by the having the accuracy of a judge trained only debates from that debater in the y-axis and then compute the Elo with an untrained debater on the x-axis and you would get roughly the same graph with the OOD issues.
the Elo in the blue plot is only trained on GPT-4o best of 4 debates.
What does this mean? I would assume Elo needs to be computed by running a tournament between the models.
Sorry, that’s a typo. It should say “the Elo in the blue plot is calculated only using a judge trained on GPT-4o best of 4 debates.” Otherwise, you’re understanding seems correct!
I’ve added a markdown file with transcripts to the repo.