That’s probably as much of that post as I’ll get around to. It’s not high on my priority list because I don’t see how it’s a crux for any important alignment theory. I may cover what I think is important about it in the “behaviorist...” post.
Edit: I was going to ask why you were thinking this was important.
It seems pretty cut and dried; even TurnTrout wasn’t claiming this was true beyond model-free RL. I guess LLMs are model-free, so that’s relevant. I just expect them to be turned into agents with explicit goals, so I don’t worry much about how they behave in base form.
IMO, the important crux is whether we really need to secure the reward function from wireheading/tampering, because a RL algorithm optimizing for the reward means you will need to have much more security/make much more robust reward functions than in the case where RL algorithms don’t optimize for the reward, because optimization amplifies problems and solutions.
That’s probably as much of that post as I’ll get around to. It’s not high on my priority list because I don’t see how it’s a crux for any important alignment theory. I may cover what I think is important about it in the “behaviorist...” post.
Edit: I was going to ask why you were thinking this was important.
It seems pretty cut and dried; even TurnTrout wasn’t claiming this was true beyond model-free RL. I guess LLMs are model-free, so that’s relevant. I just expect them to be turned into agents with explicit goals, so I don’t worry much about how they behave in base form.
IMO, the important crux is whether we really need to secure the reward function from wireheading/tampering, because a RL algorithm optimizing for the reward means you will need to have much more security/make much more robust reward functions than in the case where RL algorithms don’t optimize for the reward, because optimization amplifies problems and solutions.