In a previous post, I asked which impact measures were vulnerable to subagents. Vika pointed out that it was not merely an issue of of the impact measure, but also of the baseline.
This is indeed the case, but the nature of the impact measure is still relevant. In this post, I’ll establish two facts: that under the stepwise inaction baseline, a subagent completely undermines all impact measures (including twenty billion questions). And for the inaction baseline, for non-indexical impact measures, a subagent will not change anything.
The next post will delve into the very interesting things that happen with the inaction baseline and an indexical impact measure.
Stepwise inaction baseline
All impact measures are some form of distance measure between two states, of the form ρ(st,s′t), where st is the state the agent is actually in, and s′t is some baseline state to compare with.
For the stepwise inaction baseline, s′t is calculated as follows. Let st−1 be the previous state, at−1 the previous action. Then s′t is the state that would have followed st−1 had the agent taken the noop action, ∅, instead of at−1.
However, in the presence of a subagent, the agent merely has to always take the action at−1=∅. In that case, st=s′t (in a deterministic setting[1]), and the penalty ρ(st,s′t) collapses to nothing.
This leaves the subagent free to maximise R0 (the positive reward for agent and subagent), without any restrictions beyond making sure that the agent can always take the ∅ action.
Non-indexical impact penalty
The twenty billion questions is an example of a non-indexical impact penalty. The impact penalty is defined as the difference in expected value of many variables, between the current state and the baseline state.
With the inaction baseline state, s′t is the state the system would have been in, had the agent always done nothing/had not been turned on. Thus the definition of s′t is independent of the agent’s actions, so the inaction baseline does not collapse like the stepwise inaction baseline does.
What about subagents? Well, since the impact penalty is non-indexical, a subagent does not get around it. It matters not whether the subagent is independent, or an extension of the agent: the impact penalty remains.
Stepwise inaction and non-indexical impact measures
Overall summary post here.
In a previous post, I asked which impact measures were vulnerable to subagents. Vika pointed out that it was not merely an issue of of the impact measure, but also of the baseline.
This is indeed the case, but the nature of the impact measure is still relevant. In this post, I’ll establish two facts: that under the stepwise inaction baseline, a subagent completely undermines all impact measures (including twenty billion questions). And for the inaction baseline, for non-indexical impact measures, a subagent will not change anything.
The next post will delve into the very interesting things that happen with the inaction baseline and an indexical impact measure.
Stepwise inaction baseline
All impact measures are some form of distance measure between two states, of the form ρ(st,s′t), where st is the state the agent is actually in, and s′t is some baseline state to compare with.
For the stepwise inaction baseline, s′t is calculated as follows. Let st−1 be the previous state, at−1 the previous action. Then s′t is the state that would have followed st−1 had the agent taken the noop action, ∅, instead of at−1.
However, in the presence of a subagent, the agent merely has to always take the action at−1=∅. In that case, st=s′t (in a deterministic setting[1]), and the penalty ρ(st,s′t) collapses to nothing.
This leaves the subagent free to maximise R0 (the positive reward for agent and subagent), without any restrictions beyond making sure that the agent can always take the ∅ action.
Non-indexical impact penalty
The twenty billion questions is an example of a non-indexical impact penalty. The impact penalty is defined as the difference in expected value of many variables, between the current state and the baseline state.
With the inaction baseline state, s′t is the state the system would have been in, had the agent always done nothing/had not been turned on. Thus the definition of s′t is independent of the agent’s actions, so the inaction baseline does not collapse like the stepwise inaction baseline does.
What about subagents? Well, since the impact penalty is non-indexical, a subagent does not get around it. It matters not whether the subagent is independent, or an extension of the agent: the impact penalty remains.
In a non-deterministic setting, ρ(st,s′t) becomes a mix of an impact penalty and a measure of environment stochasticity.