I think in the limit of text prediction, language models can learn ~all of humanity’s shared world model that is represented explicitly. The things that language models can’t learn are IMO:
Tacit knowledge of the world that we haven’t represented in text
Underdetermined features of the world
Aspects of our shared world model as represented in language that do not uniquely constrain our particular universe
I think in the limit of text prediction, language models can learn ~all of humanity’s shared world model that is represented explicitly. The things that language models can’t learn are IMO:
Tacit knowledge of the world that we haven’t represented in text
Underdetermined features of the world
Aspects of our shared world model as represented in language that do not uniquely constrain our particular universe
Stuff we don’t know about the world