I think what Dan is saying is not “There could be certain intelligent behaviours present during training that disappear during inference.” The point as I understand it is “Because GPT does not learn long-term from prompts you give it, the intelligence it has when training is finished is all the intelligence that particular model will ever get.”
I think what Dan is saying is not “There could be certain intelligent behaviours present during training that disappear during inference.” The point as I understand it is “Because GPT does not learn long-term from prompts you give it, the intelligence it has when training is finished is all the intelligence that particular model will ever get.”