so while I agree with your assertion of possibility, it still seems to me that extracting signal from noise is the hard part, and additional data is needed for any agent to decide which behaviors online it’d like to imitate. if that feedback data is, eg, RLHF, then while the overall agent can be stronger than human—potentially much stronger—its ability to improve will be limited to the intelligence of the RLHF feedback process.
but currently, you don’t get human level chess play even training on a good dataset of chess moves unless you also do something like add a reinforcement learner: Modeling Strong and Human-Like Gameplay with KL-Regularized Search
so while I agree with your assertion of possibility, it still seems to me that extracting signal from noise is the hard part, and additional data is needed for any agent to decide which behaviors online it’d like to imitate. if that feedback data is, eg, RLHF, then while the overall agent can be stronger than human—potentially much stronger—its ability to improve will be limited to the intelligence of the RLHF feedback process.