Double Corrigibility: better Corrigibility

[Stuart_Armstrong]

This post is obsolete, replaced by this one.

EDIT: Defining $C$ and the conditions on $C$ is a bit more complicated than presented here, still have some details to formalise.

Corrigibility was an attempt to allow an AI to safely change its values, without seeking to provoke or avoid the change. The idea is that, when the AI’s utility changes from $u$ to $v$ at time $t$, it maximises a meta-utility $U$ such that

• $U_{\le t}=u$.

• $U_{>t}=v+E(u|u\to u)-E(v|u\to v)$.

---

Here $u\to u$ designates the event that $u$ remains unchanged, while $u\to v$ designates the change.

As has been shown, adding those expectation terms means the AI will not seek to provoke or resist such a utility change—it is indifferent to the change. Those expectation terms can be designated compensatory rewards.

What it is not indifferent to, though, is changes to $E$. If the AI knows that change $u\to v$ is coming, then it has incentives to change $E$ to $E^{\prime }$ with a high $E^{\prime }(u|u\to u)$ and a low $E^{\prime }(v|u\to v)$. Its ability to rewire is limited by the fact that $E^{\prime }$ will serve as its future expectation estimator, so it won’t mangle it completely, but it certainly can twist it a bit.

Notice that the AI is now no longer indifferent to the change of utility. It can benefit from rewiring $E$ this way only if a change is coming up. Notice that $E^{\prime }$ probably can’t rewire itself any more, so the rewired AI will be indifferent, but not before rewiring.

Double corrigibility

To solve this problem, we need to turn to corrigibility a second time. Let $\rho$ be the AI’s current expectation/​probability estimator. Let ${\rho }^{\prime }$ be one that it can conceivably change into. Let $C$ be a random variable designating the sum total of all compensatory rewards. Now, to guarantee that $C$ is finite would require bounds on all the utility functions and some discount rate. In practice, if often will be finite. That’s because $E\left(C\right)=E(u_0|u_0\to u_0)-E(u_n|u_{n-1}\to u_n)$, where $u_0$ is the AI’s first utility and $u_n$ its last. So some milder restrictions on the $u_i$ should suffice to bound $C$.

Then, whenever the AI takes an action that changes $\rho$ to ${\rho }^{\prime }$, it gets the special meta-compensatory reward

• $E_{\rho }(C|\rho \to \rho )-E_{\rho }(C|\rho \to {\rho }^{\prime })$.

This prevents it from deriving any advantage to $C$ via changes to $\rho$. Or, more precisely, it prevents it from deriving any expected advantage, that it can figure out using its current $\rho$.

Note that it doesn’t prevent improvements to $\rho$ in general, it just removes the incentives to change it to affect compensatory rewards.

Since any action $a$ might have an indirect effect on $\rho$, the “utility maximising” for a given $u$ must be changed to:

• $E_{\rho }\left(u|a\right)+{\sum }_{{\rho }^{\prime }}{P}_{\rho }(\rho \to {\rho }^{\prime }|a)\left(E_{\rho }\right(C|\rho \to \rho )-E_{\rho }(C|\rho \to {\rho }^{\prime }\left)\right)$,

where $P_{\rho }$ is the probability estimate corresponding to $\rho$; the probability term can be rewritten as $E_{\rho }\left(I_{\rho \to {\rho }^{\prime }}\right)$ for $I_{\rho \to {\rho }^{\prime }}$ the indicator function for $\rho \to {\rho }^{\prime }$. In fact the whole line above can be rewritten as

• $E_{\rho }\left(u|a\right)+E_{\rho }\left(E_{\rho }\right(C|\rho \to \rho )-E_{\rho }(C|\rho \to {\rho }^{\prime }\left)|a\right)$.

For this to work, $E_{\rho }$ needs to be able to say sensible things about itself, and also about $E_{{\rho }^{\prime }}$, which is used to estimate $C$ if $\rho \to {\rho }^{\prime }$.

If we compare this with various ways of factoring out variables, we can see that it’s a case where we have a clear default, $\rho$, and are estimating deviations from that.