As far as I understand, Gary Marcus argues that LLMs memorize some of their training data, while Hinton argues that no such thing takes place, it’s all just patterns of language.
Am I missing something here? Are these two positions compatible or does one need to be wrong for the other one to be correct? What is the crux between them and what experiment could be devised to test it?
[Question] What experiment settles the Gary Marcus vs Geoffrey Hinton debate?
It started with this video of Hinton taking a jab at Marcus: https://twitter.com/tsarnick/status/1754439023551213845
And here is Marcu’s answer:
https://garymarcus.substack.com/p/deconstructing-geoffrey-hintons-weakest
As far as I understand, Gary Marcus argues that LLMs memorize some of their training data, while Hinton argues that no such thing takes place, it’s all just patterns of language.
I found these two papers on LLM memorization:
https://arxiv.org/abs/2202.07646 - Quantifying Memorization Across Neural Language Models
https://browse.arxiv.org/abs/2311.17035 - Scalable Extraction of Training Data from (Production) Language Models
Am I missing something here? Are these two positions compatible or does one need to be wrong for the other one to be correct? What is the crux between them and what experiment could be devised to test it?