Though, as I noted in a separate comment, I agree with the basic arguments in “Understanding of capabilities is valuable” section, one thing that I’m still a bit worried about in the context of the ARC report explicitly is that labs might try to compete with each other on doing the “best” they can on the ARC eval to demonstrate that they have the most capable model, which seems probably bad (though it is legitimately unclear whether this is actually bad or not).
However, if it is really bad, here’s an idea: I think you could avoid that downside while still capturing the upside of making it clear publicly how capable models are (e.g. for the purpose of galvanizing policy responses) by revealing only the max performance on each task across all the evaluated models, rather than revealing the results individually for each model.
What we’ve currently published is ‘number of agents that completed each task’, which has a similar effect of making comparisons between models harder—does that seem like it addresses the downside sufficiently?
Though, as I noted in a separate comment, I agree with the basic arguments in “Understanding of capabilities is valuable” section, one thing that I’m still a bit worried about in the context of the ARC report explicitly is that labs might try to compete with each other on doing the “best” they can on the ARC eval to demonstrate that they have the most capable model, which seems probably bad (though it is legitimately unclear whether this is actually bad or not).
However, if it is really bad, here’s an idea: I think you could avoid that downside while still capturing the upside of making it clear publicly how capable models are (e.g. for the purpose of galvanizing policy responses) by revealing only the max performance on each task across all the evaluated models, rather than revealing the results individually for each model.
What we’ve currently published is ‘number of agents that completed each task’, which has a similar effect of making comparisons between models harder—does that seem like it addresses the downside sufficiently?
plus-one-ing the impulse to “look for third options”