Intuitively, it can be that language contains the schemes of human thought, not just as that abstract thing which produced the stream of language, but within the language itself, even though we did not lay down explicitly the algorithm of a human in words. If imitation training can find associations that somehow tap into this recursiveness, it could be that optimizing the imitation of a relatively short amount of human text was sufficient to crack humans.
This is well said. When does acting becomes indistinguishable from reality? In human world, we certainly have plenty of examples—movie actors, politicians, fake-it-till-you-make-it entrepreneurs. And more frequently, thinking out loud is something many of us practice, where the spoken words do seem to take on their own lives in pushing that abstract thing called thinking.
This is reinforcement learning, and it worked out spectacularly for AlphaGo (having to operate in a much greater search space than chess, BTW). In more constrained problem spaces, which in my mind include most of “knowledge work” / desk jobs, the amount of labeled data needed seems to be in the order of 00s of 000s.