Overall I think this is a good post and very interesting, thanks.
I find this somewhat compelling, but less so than I used to, since I’ve realized that the line between imitation learning and reinforcement learning is blurrier than I used to think (e.g. see this or this).
So I checked out those links. Briefly looking at them, I can see what you mean about the line between RL and imitation learning being blurry. The first paper seems to show a version of RL which is basically imitation learning.
I’m confused because when you said this makes iterated amplification less compelling to you, I took that to mean it made you less optimistic about iterated amplification as a solution for alignment. But why would whether something is technically classified as imitation learning or a special kind of RL make a difference for its effectiveness?
Or did you mean not that you find it any less promising as an alignment proposal, but just that you now find the core insight less compelling/interesting because it’s not as major an innovation over the idea of RL as you had thought it was?
Overall I think this is a good post and very interesting, thanks.
So I checked out those links. Briefly looking at them, I can see what you mean about the line between RL and imitation learning being blurry. The first paper seems to show a version of RL which is basically imitation learning.
I’m confused because when you said this makes iterated amplification less compelling to you, I took that to mean it made you less optimistic about iterated amplification as a solution for alignment. But why would whether something is technically classified as imitation learning or a special kind of RL make a difference for its effectiveness?
Or did you mean not that you find it any less promising as an alignment proposal, but just that you now find the core insight less compelling/interesting because it’s not as major an innovation over the idea of RL as you had thought it was?