Nora Belrose comments on Seriously, what goes wrong with “reward the agent when it makes you smile”?

[Nora Belrose]

This is actually a pretty good argument, and has caused me to update more strongly to the view that we should be optimizing only the thought process of chain of thought language models, not the outcomes that they produce