It might just be my inexperience with reinforcement learning, but while I agree with what you say, I can’t square it with my intuition of what a ML model does.
If our model uses some variant of gradient ascent, it will end up in high reward function values. (Not necessarily in any global/local maxima, but the attempt is to get it to some such maxima.) In that sense the model does optimize for reward.
Is that a special attribute of gradient ascent, that we shouldn’t expect other models to have? Does that mean that gradient ascent models are more dangerous? Are you just noting that the model won’t necessarily find the global maxima, and only reach some local maxima?
If our model uses some variant of gradient ascent, it will end up in high reward function values.
Agreed.
In that sense the model does optimize for reward.
Disagreed. Consider vanilla PG, which is as close as I know of to “doing gradient ascent in the reward landscape.” Here, the RL training process is optimizing the model in the direction of historically observed rewards. In such policy gradient methods, the model receives local cognitive updates (in the form of gradients) to increasing the logits on actions which are judged to have produced reward (e.g. in vanilla PG, this is determined by “was the action part of a high-reward trajectory?”). The model is being optimized in the direction of previous rewards, given the collected data distribution (e.g. put some trash away and observed some rewards) and the given states and its current paramterization.
This process might even find very high reward policies. I expect it will. But that doesn’t mean the model is optimizing for reward.
I’m feeling confused.
It might just be my inexperience with reinforcement learning, but while I agree with what you say, I can’t square it with my intuition of what a ML model does.
If our model uses some variant of gradient ascent, it will end up in high reward function values. (Not necessarily in any global/local maxima, but the attempt is to get it to some such maxima.) In that sense the model does optimize for reward.
Is that a special attribute of gradient ascent, that we shouldn’t expect other models to have? Does that mean that gradient ascent models are more dangerous? Are you just noting that the model won’t necessarily find the global maxima, and only reach some local maxima?
Agreed.
Disagreed. Consider vanilla PG, which is as close as I know of to “doing gradient ascent in the reward landscape.” Here, the RL training process is optimizing the model in the direction of historically observed rewards. In such policy gradient methods, the model receives local cognitive updates (in the form of gradients) to increasing the logits on actions which are judged to have produced reward (e.g. in vanilla PG, this is determined by “was the action part of a high-reward trajectory?”). The model is being optimized in the direction of previous rewards, given the collected data distribution (e.g. put some trash away and observed some rewards) and the given states and its current paramterization.
This process might even find very high reward policies. I expect it will. But that doesn’t mean the model is optimizing for reward.
That was my takeaway as well, but I’m also somewhat confused.