nostalgebraist comments on AI #23: Fundamental Problems with RLHF

[nostalgebraist]

Agreed. I wrote some more notes about the phenomenon here.