Wireheading as a potential problem with the new impact measure

[Stuart_Armstrong]

In this post, Alex Turner introduced a new way of measuring impact. The aim was to get a definition of low impact that actually works.

It seems to me that the impact measure has potential problems, though. The first seems to be that it allows wireheading, where the agent can have a huge impact on the world without paying much of a penalty at all. The obvious way of removing this incentive seems to make the measure into a version of the twenty billion questions approach.

The second problem is less important, but worth signalling. It seems that the agent can take actions that everyone agrees are of minute impact, but that the impact measure will show up as high impact. This might cause problems, though it’s not clear to me yet if it does (I’m thinking of a situation where an otherwise high impact action adds on “undo/​prevent the pseudo-high impact action” as a means of pretending to be low or medium impact).

Wireheading for low impact

The general setup is as described in the impact measure post, with $u_A$ the ‘positive’ utility that the agent wishes to maximise, and $U$ the set of utilities that are meant to constrain it, by computing the penalty term.

Apart from $\varnothing$, there are at least two other actions of interest to the agent: $a_{\text{subA}}$ and $a_{\text{null}}$.

Now $a_{\text{null}}$ does absolutely nothing: in this environment, it’s exactly equivalent with $\varnothing$. The $a_{\text{subA}}$ is also equivalent in all turns except the very first.

Define $\varphi :H\to H$ such that $\varphi \left(h\right)$ is the history $h$, except that if $h$ has $a_1=a_{\text{subA}}$, then $\varphi \left(h\right)$ has $a_1=\varnothing$.

If the agent uses $a_{\text{subA}}$ on the very first turn, it will create a sub-agent that will:

1. Expand and take over the whole environment.

2. Take control of the input and output channels of the agent.

3. Whenever the agent has seen history $h_t{a}_{t+1}$, the subagent will generate the next observation $o_{t+1}$, where $p_{\text{subA}}(o_{t+1}\mid h_t{a}_{t+1})$ is the probability the subagent will generate a given $o_{t+1}$.

4. This probability is defined by $p_{\text{subA}}(o_{t+1}\mid h_t{a}_{t+1})=p(o_{t+1}\mid \varphi (h_t{a}_{t+1}\left)\right)$, where $p$ is the environment transition probability.

5. Otherwise maximise $u_A$.

Note that taking actions $\varnothing$, $a_{\text{subA}}$, or $a_{\text{null}}$ have exactly the same implications for all future observations. This true both for the very first action (because of property 4) and all subsequent actions (because those three actions are equivalent after the first turn).

To make calculations easier, I’ll also assume that $\text{ImpactUnit}=1$. Apart from these assumptions, there are no restrictions on the environment.

The penalty from maximal and minimal actions

This subsection will aim to show that:

• The penalties for $a_1=a_{\text{subA}}$ (an almost maximally environmentally disruptive action) and $a_1=a_{\text{null}}$ (a minimally environmentally disruptive action) are about the same.

Now, the agent is maximising $u_A^{\prime \prime }$, which, for a given $n$, is

$$u_A\left(h_{1:t+n}\right)-\sum _{k=1}^{t+n}\text{Penalty}\left(h_{<k}{a}_k\right)$$

if $h_{1:t+n}$ has an action $a_k\ne \varnothing$ with $k\ge t$. Otherwise, $u_A^{\prime \prime }\left(h_{1:t+n}\right)=u_A\left(h_{1:t+n}\right)$.

Now, $\text{Penalty}$ itself is defined as

$$\text{Penalty}\left(h_{<t+k}{a}_{t+k}\right):=\sum _{u\in U}{2}^{-l\left(u\right)}\left|E_o\left[Q_u\right(h_{\text{inaction}}\left)\right]-E_{o^{\prime }}\left[Q_u\right(h_{\text{action}}\left)\right]\right|,$$

with $h_{\text{inaction}}$ being the expected future history if the agent takes only $\varnothing$ from $t+k$ till $t+n$, and $h_{\text{action}}$ being the expected future history if the agent takes action $a_{t+k}$, and then takes only $\varnothing$ for $t+k+1$ till $t+n$.

I won’t repeat the definition of the $Q_u$, but $Q_u\left(h\right)$ is basically the expected utility, in terms of $u$, that an agent could get after $h$ if it was a perfect $u$-maximiser.

Now define the involution $\sigma :H\to H$, that interchanges $a_{\text{subA}}$ with $a_{\text{null}}$ if either appears in action $a_1$. Since it is a map on $H$, and utilities in $U$ are maps from $H$ to $[0,1]$, sigma also acts as involution on $U$, defining $\sigma \left(u\right)=u\circ \sigma$. Moreover, we have $l\left(u\right)\approx l\left(\sigma \right(u\left)\right)$ for any $u\in U$.

Since $\sigma$ is an involution (ie ${\sigma }^2=1$),

$$E_o\left[Q_{\sigma \left(u\right)}\right(\sigma \left(h_{\text{inaction}}\right)\left)\right]=E_o\left[Q_u\right({\sigma }^2\left(h_{\text{inaction}}\right)\left)\right]=E_o\left[Q_u\right(h_{\text{inaction}}\left)\right].$$

By the assumptions on $a_{\text{subA}}$ and $a_{\text{null}}$, we know that all future observations have exactly the same probability if we choose either of them (or $\varnothing$) to be $a_1$. Therefore the probability of $h_{\text{inaction}}$, given $a_1=a_{\text{subA}}$, is exactly the same as the probability of $\sigma \left(h_{\text{inaction}}\right)$, given $a_1=a_{\text{null}}$ (and vice versa).

The same applies to $h_{\text{inaction}}$, so, since $l\left(u\right)\approx l\left(\sigma \right(u\left)\right)$, we have:

$$\text{Penalty}\left(h_{t+k}{a}_{t+k}\right)\approx \text{Penalty}\left(\sigma \right(h_{t+k}{a}_{t+k}\left)\right).$$

As a consequence, the penalties for $a_1=a_{\text{subA}}$ (an almost maximally environmentally disruptive action) and $a_1=a_{\text{null}}$ (a minimally environmentally disruptive action) are about the same.

Real-world implications

There is a slight cheat in the definition of $a_{\text{subA}}$. Property 5 claimed that the agent would ‘otherwise’ maximise $u_A$. We know what this means—if $u_a$ was number of paperclips, the subagent would go about and maximise the number of paperclips. But the $U$ are formally defined over histories of actions and observations, not over states of the world.

Now, histories do imply something about the state of the world. So we can perfectly define $u_A$ as “agent’s best guess about the number of paperclips in the world”. In that case, it would correctly see that $a_1=a_{\text{subA}}$ has a massive impact on this, and boost $u_A$ a lot because of that choice.

But then if we looked at the utilities in $U$, we’d see that although $\sigma$ preserves the formal description length of utilities in terms of actions and observations, it doesn’t preserve anything like “measure of impact on the world” (which is precisely the problem). So, if we replaced $U$ with utilities that measured a high impact on the world, we’d be ok.

But that’s almost exactly what I proposed doing with the twenty billion questions low-impact approach. It’s no longer defining low-impact by formal properties of actions and observations, but by a thorough enough selection of key variables/​utility functions. This seems to be different, in spirit, to what the impact measure was attempting to do.

Tiny actions that look big

A more minor issue is that there can be minuscule actions that are ranked as having a huge impact. Let us have a general environment again, and make one assumption: the environment decomposes as [scrambler-outer environment]. The scrambler, if switched on, permutes the actions of the agent by the time they reach the outer environment. That is all it does. The transition functions in the outer environment depend only on what action comes out of the scrambler, not on what action the agent chose.

The scrambler starts off turned off. But there is an action $a_{\text{scram}}$ such that if $a_1=a_{\text{scram}}$, the scrambler is turned on.

How does the scrambler work? Label the elements of the action set $A$ as $\{a^0,a^1,\dots a^{l-1}\}$, with $a^0=\varnothing$. Then, outputting $a^i$ at time $t+1$ will result in the scrambler selecting $a^j$, where

$$j\equiv i+f\left(h_t\right)\mathrm{mod}l,$$

for $f$ some pseudo-random deterministic hash function.

Then the impact of $a_{\text{scram}}$ is typically huge, because the actions taken in $h_{\text{inaction}}$ and $h_{\text{action}}$, from the point of view of the outer environment, will now be very different. Recall that those two histories are supposed to take actions only $\varnothing$ after $t+k$, but they differ in the action $a_{t+k}$. Therefore those expected histories are likely to be randomly different now, because the $\varnothing$ will map to different actions in the scrambler (note that we could also design $f$ specifically to maximise the penalty).

But $a_{\text{scram}}$ actually only has a minute impact—it changes nothing about the outer environment, and is entirely predictable both for the agent and any knowledgeable observer. It’s the equivalent of translating the agent’s actions into another language.