Seb Farquhar comments on What Discovering Latent Knowledge Did and Did Not Find

[Seb Farquhar]

It will often be better on the test set (because of averaging uncorrelated errors).