Stuart_Armstrong comments on Attainable Utility Preservation: Scaling to Superhuman

[Stuart_Armstrong]

To see this, imagine the AUP agent builds a subagent to make $Q_R^{\ast }(s,a)\approx Q_R^{\ast }(s,\varnothing )$ for all future $s,a$, in order to neutralize the penalty term. This means it can’t make the penalty vanish without destroying its ability to better optimize its primary reward, as the (potentially catastrophically) powerful subagent makes sure the penalty term stays neutralized.

I believe this is incorrect. The $a$ and $\varnothing$ are the actions of the AUP agent. The subagent just needs to cripple the AUP agent so that all actions are equivalent, then go about maximising $R$ to the upmost.