Double indifference is better indifference

[Stuart_Armstrong]

A putative new idea for AI control; index here.

Note: this post replaces and supersedes the previous one which is now obsolete.

Hacking utility indifference

Utility indifference was a technique for corrigibility was an attempt to allow an AI to safely change its values, without seeking to provoke or avoid the change. The idea is that, when the AI’s utility changes from $u$ to $v$ at time $t$, it maximises a meta-utility $U$ such that

• $U_{\le t}=u$.

• $U_{>t}=v+E(u|u\to u)-E(v|u\to v)$.

---

Here $u\to u$ designates the event that $u$ remains unchanged, while $u\to v$ designates the change. This is a permitted utility change—one that we have set the agent up to accept—rather than just any old change in utility, which the agent will still attempt to resist.

As has been shown, adding those expectation terms means the AI will not seek to provoke or resist such a utility change—it is indifferent to the change. Those expectation terms can be designated compensatory rewards.

What it is not indifferent to, though, is changes to $E$. If the AI knows that change $u\to v$ is coming, then it has incentives to change $E$ to $E^{\prime }$ with a high $E^{\prime }(u|u\to u)$ and a low $E^{\prime }(v|u\to v)$. Its ability to rewire is limited by the fact that $E^{\prime }$ will serve as its future expectation estimator, so it won’t mangle it completely, but it certainly can twist it a bit.

Notice that the AI is now no longer indifferent to the permitted change of utility. It can benefit from rewiring $E$ this way only if a change in utility is coming up.

Probability estimators

Let ${\rho }_i$ be probability estimators. These define both expectation estimators $E_{{\rho }_i}$ and probability estimators $P_{{\rho }_i}$. These two are equivalent in principle, since for any random variable $X$ with $I_x$ the indicator function for $X=x$, then $P_{{\rho }_i}(X=x)=E_{{\rho }_i}{I}_x$ and $E_{{\rho }_i}X={\sum }_x{P}_{{\rho }_i}(X=x)$.

Double indifference

One idea to combat this is to apply indifference to the process of changing probability estimators. Thus an AI with estimator ${\rho }_i$ changing to ${\rho }_j$ would get compensatory reward

• $E_{{\rho }_i}(U|{\rho }_i\to {\rho }_i)-E_{{\rho }_i}(U|{\rho }_i\to {\rho }_j)$.

This won’t work, however, since we don’t want to discount all the impacts of the change ${\rho }_i\to {\rho }_j$ (maybe ${\rho }_j$ is a much more accurate estimator), just the part of the change that affects the compensatory rewards. So the second layer of indifference will be a bit different from the previous one.

First, a subtlety. If we designate by $\rho ={\rho }_i$ the fact that the agent uses estimator ${\rho }_i$, it is very possible that $E_{{\rho }_i}(u|\rho ={\rho }_j)>E_{{\rho }_i}(u|\rho ={\rho }_i)$. This simply means that, in ${\rho }_i$‘s ‘opinion’, ${\rho }_j$ is a better estimator for maximising $u$ than it is itself.

Now, assume it’s at a time $t$ where the agent is using estimator ${\rho }_t$ and current utility $u_t$. Let $C_{>t}(u,v,{\rho }^{\prime })$ be all future compensatory rewards of the type $E_{{\rho }^{\prime }}(u|u\to u,\rho ={\rho }^{\prime })-E_{{\rho }^{\prime }}(v|u\to v,\rho ={\rho }^{\prime })$, for permitted change $u\to v$. Let $C_{>t}(u,v,{\rho }^{\prime \prime },{\rho }^{\prime })$ be the same, except the expectation estimator is ${\rho }^{\prime \prime }$: thus it is $E_{{\rho }^{\prime \prime }}(u|u\to u,\rho ={\rho }^{\prime })-E_{{\rho }^{\prime \prime }}(v|u\to v,\rho ={\rho }^{\prime })$. Note that $C_{>t}(u,v,{\rho }^{\prime },{\rho }^{\prime })=C_{>t}(u,v,{\rho }^{\prime })$.

Then define $C_{>t}$ (and $C_{>t}\left({\rho }^{\prime \prime }\right)$) by summing $C_{>t}(u,v,{\rho }^{\prime })$ (and $C_{>t}(u,v,{\rho }^{\prime \prime },{\rho }^{\prime })$) over all $u$, $v$, and ${\rho }^{\prime }$.

Let $D_{>t}$ be all future compensatory rewards that are not of the type $C_{>t}$. Then if the agent changes ${\rho }_t$ into ${\rho }^i$, it gets the compensatory reward

• $D_t=-E_{{\rho }_t}(D_{>t}|{\rho }_t\to {\rho }^i)-E_{{\rho }_t}(C_{>t}|{\rho }_t\to {\rho }^i)+E_{{\rho }_t}\left(C_{>t}\right({\rho }_t)|{\rho }_t\to {\rho }^i)$.

Note that $D_t$ is defined in terms of $D_{>t}$, so, though there may be convergence issues, the definition is not circular.

Then the first two terms of $D_t$ cancel out, according to the agent’s best current expectation, all future compensatory rewards. The third term replaces all the $C$ type compensatory rewards with the same, except estimated by the agent’s current ${\rho }_t$ rather than by whichever estimator the agent will have at the time. Thus, the agent will be indifferent to all actual compensatory rewards (as it gets none in expectation), and instead gain estimated rewards that make it indifferent to any future change $u_i\to u_j$.

See this post to see the same formula in AIXI formalism.

There are certainly cases where the above terms converge (such as all bounded $u_i$, finite number of possible changes to both utility and $\rho$). A full discussion of convergence will wait for another post.

Sequentially unbiased estimators

Call the estimators $\left\{{\rho }_i\right\}$ sequentially unbiased if for all $i<j$, and all $X$, $E_{{\rho }_i}{E}_{{\rho }_j}X=E_{{\rho }_i}X$. Then the main result is:

• If the different ${\rho }_i$ used by the agent are sequentially unbiased when labelled sequentially, then classical corrigibility is still sufficient.

To prove this, imagine that the agent will see a single change of utility $u\to v$, and that it has the opportunity of changing ${\rho }_i$ (to ${\rho }_j$, say) exactly once before the utility change. Then there will be no $D$ term for ${\rho }_j$, and, if it does change ${\rho }_i\to {\rho }_j$, the $D$ term is:

• $D_t=-E_{{\rho }_i}(C_{>t}|{\rho }_i\to {\rho }_j)+E_{{\rho }_i}\left(C_{>t}\right({\rho }_i)|{\rho }_i\to {\rho }_j)$.

However, $E_{{\rho }_i}(C_{>t}|{\rho }_i\to {\rho }_j)=E_{{\rho }_i}\left(E_{{\rho }_j}\right(u|u\to u,\rho ={\rho }_j)-E_{{\rho }_j}(v|u\to v,\rho ={\rho }_j\left)\right)$. Because of sequential unbiasedness, this simplifies to $E_{{\rho }_i}(u|u\to u,\rho ={\rho }_j)-E_{{\rho }_i}(v|u\to v,\rho ={\rho }_j)$, which is just $E_{{\rho }_i}\left(C_{>t}\right({\rho }_i)|{\rho }_i\to {\rho }_j)$. So $D_t=0$.

We can then recurse to the change in $\rho$ just before ${\rho }_i$, and get the same result (since the future $D$ is still zero). And so on, with $D$ always being zero. Then since the formulas defining $D$ are linear, we can extend this to general environments and general utility function changes, and conclude that for sequentially unbiased ${\rho }_i$, the $D$ are always $0$ under double indifference (modulo some convergence issues not addressed here). Therefore, double indifference will work, even if we don’t use $D$’s at all: thus classical indifference still works in this case.

Note the similarity of the sequential unbiasness with the conditions for successful value learners in the Cake or Death problem.

[jessicata]

I thought the main problem with utility indifference was the incentive to “gerrymander” probability mass (explained in section 4.2 of the corrigibility paper); is this addressed in either the original utility indifference proposal or the modification explained in this post?

To be more specific: one version of the approach outlined here has the AI maximize

$$v\left(\pi \right):=\sum _{b}q\left(b\right)E\left[U_b|do\right(\pi ),B=b]$$

Suppose there are 2 B values, p and s. $U_p$ says the AI should maximize paperclips; $U_s$ says the AI should maximize staples. $B$ is set to $p$ by default but there’s a button that switches it to $s$.

Assume $q$ is uniform over the 2 possible values $p$ and $s$. In the first time step, the AI gets a number of resources as an integer uniformly distributed between 1 and 10. In the second time step, the AI may manipulate $B$ by pressing the paperclips-to-staples button. In the third time step, the AI may make a number of objects (paperclips or staples) equal to the number of resources.

The optimal policy given the value function above is to observe the number of resources, press the button iff the number of resources is 10, and then creates the maximum-allowed number of copies of the object corresponding to the resulting $B$ value. This “gerrymanders” probability mass by making it so the AI gets 10 resources “half the time” instead of ¹⁄₁₀ of the time.

The only way of dealing with this problem that I know of is to use a causal counterfactual $E\left[U_b|do\right(\pi ),do(B=b\left)\right]$ instead of a conditional. I don’t understand classical utility indifference as well as I understand the factoring-out approach, but it seems like in your post you aren’t doing a causal counterfactual, so either your approach solves the gerrymandering problem in a novel way, it still has this problem, or I’m confused about something.

[Stuart_Armstrong]

Hum, I seem to have not explained very well.

In any case, the original proposal (or slight modifications of it) should not have the gerrymandering problem. Let $r$ be the number of resources. When the button is pressed, the AI gains $E[U_p|U_b=U_p,r=n]$ and loses $E[U_s|U_b=U_s,r=n]$, for some $n$. The AI is then left to maximise $U_b$, given that $U_b=U_s$ and $r=n$. Once the rewards and punishments are taken into account, the expected utility is the same whether or not the button is pressed. Since this is true for all $n$, there is no advantage to gerrymandering $n$’s value.

The factoring out variables is not, as far as I can tell, the same things as indifference via compensatory rewards.

Now, the problem with the initial indifference is that there is no incentive to gerrymander probability mass, but there is an incentive to gerrymander the probability estimator. Since the rewards depend on $E$, modifying that will earn higher rewards (and since the true $E$ is going to be a bounded approximation anyway, changes to $E$ are not inconceivable). The above post intends to deal with that problem.

[paulfchristiano]

I had this confusion long ago as well. I think the definition is much clearer if you just say “When we press the button, we flip a coin that comes up heads 1/​billion times. We only change the agent’s values /​ turn it off if the coin comes up tails, which almost always happens. The agent chooses a policy assuming that the coin comes up heads.”

[jessicata]

I still think the original proposal has a gerrymandering problem (slightly different from the one in my comment).

Say the AI is going to construct a sub-agent B (and the original AI will then go to sleep for the rest of the game, unable to view anything that happens). In the first turn, B will get a number of resources between 1 and 10. These resources are only useful for making paperclips. In the second turn, B can press a paperclips-to-staples button, which changes the original AI’s utility function to staple-maximization. In the third turn, B may create a number of paperclips equal to the number of resources, and may independently create 42 paperclips.

The naive strategy is to create a subagent that doesn’t press the button, then makes paperclips. This policy gets a reward of 5.5 (the average number of resources).

Consider an alternative strategy. The AI creates a sub-agent B with the following policy: if B gets between 1 and 5 resources, then it presses the paperclips-to-staples button; otherwise it doesn’t press the button. Then B will create the appropriate number of paperclips, or 42 staples, depending on the button.

Let u be number of paperclips produced, v be number of staples produced. Under the policy, we have $E[u|u\to u]=8$. And $E[v|u\to v]=5$. Since the button gets pressed half the time, the AI’s expected reward is

$$P(u\to u)E[u|u\to u]+P(u\to v)\left(E\right[v|u\to v]+E[u|u\to u]-E[v|u\to v\left]\right)=1/2\cdot 8+1/2(42+8-42)=8$$

Meanwhile, the AI’s expected reward if it never presses the button and just makes paperclips is 5.5. So it seems that the original proposal has a gerrymandering problem.

It’s possible that your new proposal fixes this problem, but first I want to check that this is actually a problem for the original proposal.

[Stuart_Armstrong]

This is my punishment for being sloppy with definitions :-(

Ok, let’s be clearer. When the button is pressed, a process begins, to change $u$ into $v$. But the process is stocastic, and has a small chance of failing. The failure of this is designated by $u\to u$.

So if Pr refers to pressing the button, the expected reward is:

$P\left(\neg Pr\right)E\left(u|\neg Pr\right)+P(Pr,u\to u)E(u|Pr,u\to u)+P(Pr,u\to v)\left(E\right(v|Pr,u\to v)+E(u|Pr,u\to u)-E(v|Pr,u\to v\left)\right)=P\left(\neg Pr\right)E\left(u|\neg Pr\right)+\left(P\right(Pr,u\to u)+P(Pr,u\to v\left)\right)E(u|Pr,u\to u)=\left(1/2\right)3+\left(1/2\right)8=\mathrm{5.5.}$

[jessicata]

This makes more sense, thanks for explaining!

I’m going to translate this to my notation so I understand it better (feel free to skip this, it’s mostly for my benefit). Let $X$ be the event that the coin flip succeeds (which usually happens). We can say that $X$ still has a value even if $\neg Pr$ (its value just doesn’t do anything). So $u\to v$ happens iff $Pr\wedge X$.

The expected utility is then

$$E\left[\right[\neg Pr\vee \neg X]u+[Pr\wedge X\left]\right(v+E[u|Pr\wedge \neg X]-E[v|Pr\wedge X])$$

Suppose the AI never updates its beliefs (because it goes into stasis the first round). Then this is

$$E\left[\right[\neg Pr\vee \neg X\left]u\right]+E\left[\right[Pr\wedge X\left]v\right]+P(Pr\wedge X)E[u|Pr\wedge \neg X]-E\left[\right[Pr\wedge X\left]v\right]$$ $$=E\left[\right[\neg Pr\vee \neg X\left]u\right]+P(Pr\wedge X)E[u|Pr\wedge \neg X]$$ $$=P\left(\neg Pr\right)E[u|\neg Pr,\neg X]+P(Pr\wedge X)E[u|Pr,\neg X]$$ $$\approx P\left(\neg Pr\right)E[u|\neg Pr,\neg X]+P\left(Pr\right)E[u|Pr,\neg X]$$ $$=E\left[u|\neg X\right]$$ $$=E\left[u|do\right(\neg X\left)\right]$$ (I wrote it as a causal counterfactual in the last step; it doesn’t make a difference since $X$ has no causal ancestors, but I find it easier to reason about this way)

So in the end, if the whole game is to construct a subagent and then go into stasis, then the agent is just a $u$-maximizer who believes $\neg X$ (as you said). It avoids the gerrymandering problem by doing a causal counterfactual.

I think another way of framing this is that the button always succeeds, and agent optimizes $E\left[u|do\right(\neg Pr\left)\right]$ (which relates this back to the causal version of factoring out effects, where you optimize $q\left(Pr\right)E\left[v|do\right(Pr\left)\right]+q\left(\neg Pr\right)E\left[u|do\right(\neg Pr\left)\right]$; you get utility indifference with stasis by setting $q\left(Pr\right)=0$). This will be slightly different in that here the agent “believes” it will be magically prevented from pressing the button, whereas originally the agent just “believes” that the button will fail.

[Stuart_Armstrong]

Note there is one way in which the AI functions as an outcome pump: it will accept bets, at any odds, against ever becoming a $v$-maximiser. That’s because that’s what a pure $u$-maximiser would do, and this agent design behave like a pure $u$-maximiser.

[orthonormal]

Typo: in the paragraph before the equation arrays, you forgot to change $E[v|u\to v]$ from 5 to 42 (you did so in the following equation arrays). This buffaloed me for a bit!

[jessicata]

Fixed, thanks.

[jessicata]

The idea is that, when the AI’s utility changes from u to v at time t, it maximises a meta-utility U

What does it mean for the meta-utility U to depend on the time step t? My understanding is that utility functions are over world histories; thus it doesn’t make sense for them to depend on the time step.

My guess is that you meant that both $u$ and $v$ are expressed as a sum of rewards over time, and the meta-utility sums the rewards of $u$ before $t$ with the rewards of $v$ after $t$ (plus an expected reward correction term); is this correct?