Double indifference is better indifference

[Stuart_Armstrong]

A putative new idea for AI control; index here.

Note: this post replaces and supersedes the previous one which is now obsolete.

Hacking utility indifference

Utility indifference was a technique for corrigibility was an attempt to allow an AI to safely change its values, without seeking to provoke or avoid the change. The idea is that, when the AI’s utility changes from $u$ to $v$ at time $t$, it maximises a meta-utility $U$ such that

• $U_{\le t}=u$.

• $U_{>t}=v+E(u|u\to u)-E(v|u\to v)$.

---

Here $u\to u$ designates the event that $u$ remains unchanged, while $u\to v$ designates the change. This is a permitted utility change—one that we have set the agent up to accept—rather than just any old change in utility, which the agent will still attempt to resist.

As has been shown, adding those expectation terms means the AI will not seek to provoke or resist such a utility change—it is indifferent to the change. Those expectation terms can be designated compensatory rewards.

What it is not indifferent to, though, is changes to $E$. If the AI knows that change $u\to v$ is coming, then it has incentives to change $E$ to $E^{\prime }$ with a high $E^{\prime }(u|u\to u)$ and a low $E^{\prime }(v|u\to v)$. Its ability to rewire is limited by the fact that $E^{\prime }$ will serve as its future expectation estimator, so it won’t mangle it completely, but it certainly can twist it a bit.

Notice that the AI is now no longer indifferent to the permitted change of utility. It can benefit from rewiring $E$ this way only if a change in utility is coming up.

Probability estimators

Let ${\rho }_i$ be probability estimators. These define both expectation estimators $E_{{\rho }_i}$ and probability estimators $P_{{\rho }_i}$. These two are equivalent in principle, since for any random variable $X$ with $I_x$ the indicator function for $X=x$, then $P_{{\rho }_i}(X=x)=E_{{\rho }_i}{I}_x$ and $E_{{\rho }_i}X={\sum }_x{P}_{{\rho }_i}(X=x)$.

Double indifference

One idea to combat this is to apply indifference to the process of changing probability estimators. Thus an AI with estimator ${\rho }_i$ changing to ${\rho }_j$ would get compensatory reward

• $E_{{\rho }_i}(U|{\rho }_i\to {\rho }_i)-E_{{\rho }_i}(U|{\rho }_i\to {\rho }_j)$.

This won’t work, however, since we don’t want to discount all the impacts of the change ${\rho }_i\to {\rho }_j$ (maybe ${\rho }_j$ is a much more accurate estimator), just the part of the change that affects the compensatory rewards. So the second layer of indifference will be a bit different from the previous one.

First, a subtlety. If we designate by $\rho ={\rho }_i$ the fact that the agent uses estimator ${\rho }_i$, it is very possible that $E_{{\rho }_i}(u|\rho ={\rho }_j)>E_{{\rho }_i}(u|\rho ={\rho }_i)$. This simply means that, in ${\rho }_i$‘s ‘opinion’, ${\rho }_j$ is a better estimator for maximising $u$ than it is itself.

Now, assume it’s at a time $t$ where the agent is using estimator ${\rho }_t$ and current utility $u_t$. Let $C_{>t}(u,v,{\rho }^{\prime })$ be all future compensatory rewards of the type $E_{{\rho }^{\prime }}(u|u\to u,\rho ={\rho }^{\prime })-E_{{\rho }^{\prime }}(v|u\to v,\rho ={\rho }^{\prime })$, for permitted change $u\to v$. Let $C_{>t}(u,v,{\rho }^{\prime \prime },{\rho }^{\prime })$ be the same, except the expectation estimator is ${\rho }^{\prime \prime }$: thus it is $E_{{\rho }^{\prime \prime }}(u|u\to u,\rho ={\rho }^{\prime })-E_{{\rho }^{\prime \prime }}(v|u\to v,\rho ={\rho }^{\prime })$. Note that $C_{>t}(u,v,{\rho }^{\prime },{\rho }^{\prime })=C_{>t}(u,v,{\rho }^{\prime })$.

Then define $C_{>t}$ (and $C_{>t}\left({\rho }^{\prime \prime }\right)$) by summing $C_{>t}(u,v,{\rho }^{\prime })$ (and $C_{>t}(u,v,{\rho }^{\prime \prime },{\rho }^{\prime })$) over all $u$, $v$, and ${\rho }^{\prime }$.

Let $D_{>t}$ be all future compensatory rewards that are not of the type $C_{>t}$. Then if the agent changes ${\rho }_t$ into ${\rho }^i$, it gets the compensatory reward

• $D_t=-E_{{\rho }_t}(D_{>t}|{\rho }_t\to {\rho }^i)-E_{{\rho }_t}(C_{>t}|{\rho }_t\to {\rho }^i)+E_{{\rho }_t}\left(C_{>t}\right({\rho }_t)|{\rho }_t\to {\rho }^i)$.

Note that $D_t$ is defined in terms of $D_{>t}$, so, though there may be convergence issues, the definition is not circular.

Then the first two terms of $D_t$ cancel out, according to the agent’s best current expectation, all future compensatory rewards. The third term replaces all the $C$ type compensatory rewards with the same, except estimated by the agent’s current ${\rho }_t$ rather than by whichever estimator the agent will have at the time. Thus, the agent will be indifferent to all actual compensatory rewards (as it gets none in expectation), and instead gain estimated rewards that make it indifferent to any future change $u_i\to u_j$.

See this post to see the same formula in AIXI formalism.

There are certainly cases where the above terms converge (such as all bounded $u_i$, finite number of possible changes to both utility and $\rho$). A full discussion of convergence will wait for another post.

Sequentially unbiased estimators

Call the estimators $\left\{{\rho }_i\right\}$ sequentially unbiased if for all $i<j$, and all $X$, $E_{{\rho }_i}{E}_{{\rho }_j}X=E_{{\rho }_i}X$. Then the main result is:

• If the different ${\rho }_i$ used by the agent are sequentially unbiased when labelled sequentially, then classical corrigibility is still sufficient.

To prove this, imagine that the agent will see a single change of utility $u\to v$, and that it has the opportunity of changing ${\rho }_i$ (to ${\rho }_j$, say) exactly once before the utility change. Then there will be no $D$ term for ${\rho }_j$, and, if it does change ${\rho }_i\to {\rho }_j$, the $D$ term is:

• $D_t=-E_{{\rho }_i}(C_{>t}|{\rho }_i\to {\rho }_j)+E_{{\rho }_i}\left(C_{>t}\right({\rho }_i)|{\rho }_i\to {\rho }_j)$.

However, $E_{{\rho }_i}(C_{>t}|{\rho }_i\to {\rho }_j)=E_{{\rho }_i}\left(E_{{\rho }_j}\right(u|u\to u,\rho ={\rho }_j)-E_{{\rho }_j}(v|u\to v,\rho ={\rho }_j\left)\right)$. Because of sequential unbiasedness, this simplifies to $E_{{\rho }_i}(u|u\to u,\rho ={\rho }_j)-E_{{\rho }_i}(v|u\to v,\rho ={\rho }_j)$, which is just $E_{{\rho }_i}\left(C_{>t}\right({\rho }_i)|{\rho }_i\to {\rho }_j)$. So $D_t=0$.

We can then recurse to the change in $\rho$ just before ${\rho }_i$, and get the same result (since the future $D$ is still zero). And so on, with $D$ always being zero. Then since the formulas defining $D$ are linear, we can extend this to general environments and general utility function changes, and conclude that for sequentially unbiased ${\rho }_i$, the $D$ are always $0$ under double indifference (modulo some convergence issues not addressed here). Therefore, double indifference will work, even if we don’t use $D$’s at all: thus classical indifference still works in this case.

Note the similarity of the sequential unbiasness with the conditions for successful value learners in the Cake or Death problem.