Tailcalled talked about this two years ago. A model which predicts text does a form of imitation learning. So it is bounded by the text it imitates, and by the intelligence of humans who have written the text. Models which predict future sensory inputs (called “predictive coding” in neuroscience, or “the dark matter of intelligence” by LeCun) don’t have such a limitation, as they predict reality more directly.
Tailcalled talked about this two years ago. A model which predicts text does a form of imitation learning. So it is bounded by the text it imitates, and by the intelligence of humans who have written the text. Models which predict future sensory inputs (called “predictive coding” in neuroscience, or “the dark matter of intelligence” by LeCun) don’t have such a limitation, as they predict reality more directly.