Stuart_Armstrong comments on In theory: does building the subagent have an “impact”?

[Stuart_Armstrong]

It’s the delta of that with $Q_R(s_{t+1},a_{t+1})$ that is penalised, not large changes on its own.