During the workshop presentation, Jascha said that the OpenAI will run their models on the benchmark. This suggests that there is (was?) some collaboration. But it was a half a year ago.
Just checked, the repo’s readme doesn’t mention OpenAI anymore. In the earlier versions, it was mentioned like this:
Teams at Google and OpenAI have committed to evaluate BIG-Bench on their best-performing model architectures
So, it seems that OpenAI withdrew from the project, partially or fully.
What makes you say BIG Bench is a joint Google / OpenAI project? I’m a contributor to it and have seen no evidence of that.
During the workshop presentation, Jascha said that the OpenAI will run their models on the benchmark. This suggests that there is (was?) some collaboration. But it was a half a year ago.
Just checked, the repo’s readme doesn’t mention OpenAI anymore. In the earlier versions, it was mentioned like this:
So, it seems that OpenAI withdrew from the project, partially or fully.
OpenAI is still running evaluations.
Interesting… I was busy and wasn’t able to watch the workshop. That’s good to know, thanks!