Rohin Shah comments on Attainable Utility Preservation: Scaling to Superhuman

[Rohin Shah]

It depends whether the agent has any way of predicting what the random action will be at a future point in time.

You don’t have to literally sample a random action; you can just calculate the expected thing that would happen under a random policy. For example, you would replace $Q^{\ast }(s,\varphi )$ with $\frac{1}{|A|}\sum _{i=1}^A{Q}^{\ast }(s,a_i)$.