It doesn’t bother me that Epoch took money from OpenAI. It doesn’t bother me that OpenAI has access to the FrontierMath solutions.
What does bother me is Epoch concealing this information. I certainly assumed FrontierMath was a private eval. Clearly there are people who would not have worked on this if they’d known OpenAI would have access to the dataset. I’m really not sure why Epoch or OpenAI think misleading people about this is beneficial to them—this information coming out now, like this, just means people won’t trust Epoch in the future. Was the data they received via deception from people who wouldn’t have participated really worth burning trust like this?
I was excited about FrontierMath when it was revealed, doubly so when o3 made such impressive progress. I think o3′s results are probably uncontaminated, it would be a very bad move for OpenAI to make fake progress when they could instead make real progress, but concealing this was also a bad move so I don’t know. I really hope Epoch doesn’t pull anything like this with their upcoming computer use benchmark.
(...and I’m shocked they’re trusting verbal agreements from OpenAI about how the data is being used. Is getting stuff in writing really that hard?)
It doesn’t bother me that Epoch took money from OpenAI. It doesn’t bother me that OpenAI has access to the FrontierMath solutions.
What does bother me is Epoch concealing this information. I certainly assumed FrontierMath was a private eval. Clearly there are people who would not have worked on this if they’d known OpenAI would have access to the dataset. I’m really not sure why Epoch or OpenAI think misleading people about this is beneficial to them—this information coming out now, like this, just means people won’t trust Epoch in the future. Was the data they received via deception from people who wouldn’t have participated really worth burning trust like this?
I was excited about FrontierMath when it was revealed, doubly so when o3 made such impressive progress. I think o3′s results are probably uncontaminated, it would be a very bad move for OpenAI to make fake progress when they could instead make real progress, but concealing this was also a bad move so I don’t know. I really hope Epoch doesn’t pull anything like this with their upcoming computer use benchmark.
(...and I’m shocked they’re trusting verbal agreements from OpenAI about how the data is being used. Is getting stuff in writing really that hard?)