the neural network is capable of implementing AIs that are goal-oriented enough to want to perform well on training to prevent the training from changing them and their goals;
there’s optimization pressure in that direction: AIs like that perform better than some other AIs (which arguably won’t really be the case if your training loss is only about predicting the next token, but will be the case if you do RL in settings where advanced agency is useful).
No- only two requirements:
the neural network is capable of implementing AIs that are goal-oriented enough to want to perform well on training to prevent the training from changing them and their goals;
there’s optimization pressure in that direction: AIs like that perform better than some other AIs (which arguably won’t really be the case if your training loss is only about predicting the next token, but will be the case if you do RL in settings where advanced agency is useful).