Good to ask, but I’m not sure what it would be. The code is just a linear regression I did in a spreadsheet, and eyeballing the data points, it doesn’t look like there are any patterns that a regression is missing. I tried it several different ways (comparing to different smaller models, comparing to averages of smaller models, excluding extreme values, etc.) and the correlation was always zero. Here’s the raw data:
It’s hard to know if there is some critical bug in all the results being reported in the Gopher, Chinchilla, and PaLM papers, since we don’t have access to the models, but it would surprise me if no one caught that across multiple independent teams.
I notice you are noticing confusion. Any chance either the data, or the code, has a bug?
Good to ask, but I’m not sure what it would be. The code is just a linear regression I did in a spreadsheet, and eyeballing the data points, it doesn’t look like there are any patterns that a regression is missing. I tried it several different ways (comparing to different smaller models, comparing to averages of smaller models, excluding extreme values, etc.) and the correlation was always zero. Here’s the raw data:
https://docs.google.com/spreadsheets/d/1Y_00UcsYZeOwRuwXWD5_nQWAJp4A0aNoySW0EOhnp0Y/edit?usp=sharing
It’s hard to know if there is some critical bug in all the results being reported in the Gopher, Chinchilla, and PaLM papers, since we don’t have access to the models, but it would surprise me if no one caught that across multiple independent teams.