Authors: Edwin Zhang, Vincent Zhu, Naomi Saphra, Anat Kleiman, Benjamin L. Edelman, Milind Tambe, Sham M. Kakade, Eran Malach.
Abstract:
Generative models are trained with the simple objective of imitating the conditional probability distribution induced by the data they are trained on. Therefore, when trained on data generated by humans, we may not expect the artificial model to outperform the humans on their original objectives. In this work, we study the phenomenon of transcendence: when a generative model achieves capabilities that surpass the abilities of the experts generating its data. We demonstrate transcendence by training an autoregressive transformer to play chess from game transcripts, and show that the trained model can sometimes achieve better performance than all players in the dataset. We theoretically prove that transcendence is enabled by low-temperature sampling, and rigorously assess this experimentally. Finally, we discuss other sources of transcendence, laying the groundwork for future investigation of this phenomenon in a broader setting.
Summary thread: https://x.com/nsaphra/status/1803114822445465824.
Some previous discussion in an alignment-relevant context: GPTs are Predictors, not Imitators.
It’s common for much simpler Statistical Prediction Rules, such as linear regression or even simpler models, to outperform experts even when they were built to predict the experts’ judgment.
Yeah, ‘transcendence’ is a truly overhyped and absurd way to label what appears to be a simple ensemble effect. Should Galton have said that the crowd ‘transcended’ in guessing the weight of a fat pig?
This isn’t even a useful motivating example for how LLMs could outperform human experts while doing offline learning, because the averaging benefit breaks down, as expected, at their high end.
Exactly. All that’s needed for “transcendence” is removing some noise.
I highly recommend the book Noise by Daniel Kahneman et al on this topic.