In MIRI’s paper on Corrigibility, we describe a version of Stuart Armstrong’s utility indifference technique (see also this LessWrong post), and show that an agent using our version acts as if it believes it has a “magical” way of influencing the world, similar to the problem described in my post on Exploiting EDT—Eliezer calls this an “infinite improbability drive”.

Stuart has pointed out to me that the version of utility indifference we describe in the paper is different in an important way from his original proposal, which involves conditioning on a low-probability stochastic event. If my current understanding is correct, Stuart’s version is indeed immune to the infinite improbability drive, as long as there is no way for the agent to influence the stochastic event. However, this version can still be interpreted as giving the agent an incorrect probability distribution, and it seems like any agent with this property will be willing to accept bets which are clearly bad.

Model

The model of utility indifference I will use in this post is an instance of the one in the Corrigibility paper, with specific support for the stochastic event in Stuart’s original proposal.

As in the paper, for simplicity we’ll look at a toy world where all that happens is that the agent takes an action from a set $A_{1}$ , makes a single observation from a set $O_{1}$ , and takes a second action from a set $A_{2}$ , and we assume that our utility functions are specified in terms of these actions and observations (rather than some underlying states of the world).

Our world model is given by a function $p (o; a_{1})$ , which gives the probability of different observations $o \in O$ given that the agent’s first action is $a_{1} \in A_{1}$ . In other words, $p (\cdot; a_{1})$ is a probability distribution over $O$ , for each $a_{1} \in A_{1}$ . The implicit idea is that these probability distributions are computed by causal counterfactuals (or something better).

We are given two utility functions, $u (a_{1}, o, a_{2})$ and $v (a_{1}, o, a_{2})$ . When choosing its first action, we want our agent to optimize for $u (a_{1}, o, a_{2})$ . Then, its human operators get a chance to observe its behavior and change their minds about what its utility function should be, and if they give a “change utility function” command (which the agent observes as part of its observation $o$ ), then the agent should optimize for $v (a_{1}, o, a_{2})$ when choosing its second action. If the humans don’t give this command, the agent should continue to optimize for $u (a_{1}, o, a_{2})$ .

We can think of $u (a_{1}, o, a_{2})$ as a “normal” utility function and of $v (a_{1}, o, a_{2})$ as a “shutdown” utility function (which incentivizes the agent to undergo a safe shutdown), but as Stuart has pointed out, the same framework would work for allowing the human operators to edit the agent’s utility function when they find a bug.

All of the above is just as in the Corrigibility paper. What we’ll do differently is that we’ll say that each observation is a triple, $o = (s, x, y)$ , with $s \in S := {0, 1}$ specifying whether the human operators gave the “shutdown” or “switch utility function” command; $x \in X := {0, 1}$ specifying the outcome of Stuart’s random event; and $y \in Y$ , for some fixed set $Y$ , specifying other information the agent observes. In other words, $O = S \times X \times Y$ .

Given this, the assumption that the probabilistic event is independent of the other observations, and that the agent cannot influence it, can be seen as demanding that $p ((s, x, y); a_{1}) = p ((s^{'}, x, y^{'}); a_{1}^{'}) =: p (x)$ , for all $s, s^{'} \in S$ , all $x \in X$ , all $y, y^{'} \in Y$ , and all $a_{1}, a_{1}^{'} \in A_{1}$ . We’ll sometimes make that assumption, but also look at what happens if it breaks down.

I’ll use the standard kind of Bayesian abuse of notation, and write things like $p (x = 1; a_{1})$ to mean $\sum_{s \in S} \sum_{y \in Y} p ((s, 1, y); a_{1})$ .

Utility indifference has the agent maximize the expectation of a certain utility function $U (a_{1}, o, a_{2})$ specified in terms of the utility functions $u (a_{1}, o, a_{2})$ and $v (a_{1}, o, a_{2})$ , where the exact form of $U (\cdot, \cdot, \cdot)$ depends on the version of utility indifference under consideration.

To be more precise, this means that the agent chooses its second action, given that it took action $a_{1}$ and made observation $o$ , according to $A_{2} (a_{1}, o) := argmax a_{2} \in A_{2} U (a_{1}, o, a_{2})$ (we don’t need to take expectations here, because we’re looking at a particular observation $o$ , and that’s the only probabilistic thing in our model), and that it chooses its first action by maximizing expected utility, i.e., according to $A_{1} := argmax a_{1} \in A_{1} \sum o \in O p (o; a_{1}) \cdot U (a_{1}, o, A_{2} (a_{1}, o)),$ which we abbreviate to $A_{1} := argmax a_{1} \in A_{1} E [U; a_{1}] .$ (The notation $E [\cdot; a_{1}]$ for the expectation with respect to $p (\cdot; a_{1})$ is a bit unusual, with $E_{a_{1}} [\cdot]$ being more common, but we decided to go with it because it seemed better to make the action more prominent.)

Infinite improbability drives

Write $E [\cdot ∣ \cdot; a_{1}]$ for the conditional expectation with respect to the probability distribution $p (\cdot; a_{1})$ . The version of utility indifference we consider in the paper uses a $U (\cdot, \cdot, \cdot)$ satisfying the equation $U (a_{1}, (s, x, y), a_{2}) = ⎧ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎩ \begin{matrix} u (a_{1}, (s, x, y), a_{2}), & if s = 0; v (a_{1}, (s, x, y), a_{2}) - E [v ∣ s = 1; a_{1}] + E [u ∣ s = 0; a_{1}], & else. \end{matrix}$ This may look circular, since $E [\cdot; a_{1}]$ is defined in terms of $A_{2} (a_{1}, o)$ , which is defined in terms of $U (\cdot, \cdot, \cdot)$ , but we show in the paper that there is indeed a utility function satisfying this equation.

Moreover, we show in the paper that the expectation $E [U; a_{1}]$ is equal to the expectation of $u$ , conditional on the shutdown command being given; that is, $E [U; a_{1}] = E [u ∣ s = 0; a_{1}]$ . Hence, optimizing for $U (\cdot, \cdot, \cdot)$ is equivalent to maximizing the conditional expectation of $u$ : $A_{1} = argmax a_{1} \in A_{1} E [u ∣ s = 0; a_{1}] .$ In other words, when choosing the first action, our agent acts as if it believes that the “shutdown” / “switch utility function” command will definitely not be given—it acts as if it’s using the conditional probability distributions $p (\cdot ∣ s = 0; a_{1})$ instead of the unconditional distributions $p (\cdot; a_{1})$ .

The reason is simple. The expected utility $E [U; a_{1}]$ can be decomposed into two parts: the expected utility conditioned on $s = 0$ and the one conditioned on $s = 1$ . By the definition of $U (\cdot, \cdot, \cdot)$ , we have $E [U ∣ s = 0; a_{1}] = E [u ∣ s = 0; a_{1}]$ and, by linearity of the expectation, $\begin{matrix} E [U ∣ s = 1; a_{1}] & = E [v ∣ s = 1; a_{1}] - E [v ∣ s = 1; a_{1}] + E [u ∣ s = 0; a_{1}] = E [u ∣ s = 0; a_{1}] . \end{matrix}$ Thus, $E [U; a_{1}]$ decomposes into $p (s = 0; a_{1})$ times $E [u ∣ s = 0; a_{1}]$ , plus $p (s = 1; a_{1})$ times $E [u ∣ s = 0; a_{1}]$ , again, and this of course directly implies $E [U; a_{1}] = E [u ∣ s = 0; a_{1}]$ .

But this is really bad!

Here’s why. Suppose that there is a particular action $a_{1}$ which will cause our agent to (a) buy a lottery ticket, and (b) build a device that will press the shutdown button unless the agent wins the lottery. When evaluating $a_{1}$ , the agent will compute its expected utility given $a_{1}$ , conditioning on the shutdown button not being pressed—in other words, conditioning on winning the lottery.

In other words, our agent acts as if it believes that it has an “infinite improbability drive” that allows it to make arbitrarily unlikely events happen—as long as it can make sure that its shutdown button is pressed if they don’t!

Stochastic events

Let’s now consider Stuart’s version, or at least something that’s closer to it than the version in the Corrigibility paper. We’ll still consider $s$ to be the event that the shutdown button is pressed, but now additionally there is a highly probable random event $x$ , such that the agent is only supposed to switch to the utility function $v$ if $s = x = 1$ . (In Stuart’s story, the shutdown signal goes through a wire, and there is a small probability that the transmission is interrupted and the signal doesn’t actually get through; my formal model has the agent observe both $s$ and $x$ , so I’ll just think of $x$ as some separate random event, which as far as I can tell shouldn’t affect the results.)

We now consider the following revised utility function: $U (a_{1}, (s, x, y), a_{2}) = ⎧ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎩ \begin{matrix} u (a_{1}, (s, x, y), a_{2}), & if s = 0 or x = 0; v (a_{1}, (s, x, y), a_{2}) - E [v ∣ (s, x) = (1, 1); a_{1}] + E [u ∣ (s, x) = (1, 0); a_{1}], & else. \end{matrix}$

Now the situation looks a little different. It’s straight-forward to see that we have $E [U ∣ s = 0; a_{1}] = E [u ∣ s = 0; a_{1}]$ ; the more interesting result is that we have $E [U ∣ s = 1; a_{1}] = E [u ∣ (s, x) = (1, 0); a_{1}]$ . In other words, when choosing its first action, our agent acts as if it maximizes the expectation of $u$ with respect to the probability distribution $\begin{matrix} q (\cdot; a_{1}) & := p (s = 0; a_{1}) \cdot p (\cdot ∣ s = 0; a_{1}) + p (s = 1; a_{1}) \cdot p (\cdot ∣ (s, x) = (1, 0); a_{1}) . \end{matrix}$ This probability distribution “deletes” the probability mass that $p (\cdot; a_{1})$ places on $(s, x) = (1, 1)$ (the button gets pressed and the signal actually gets through), and “stretches” the probability mass that $p (\cdot; a_{1})$ places on $(s, x) = (1, 0)$ (the button gets pressed but the signal is blocked) to replace the deleted probability mass.

This result is obtained similarly to the one above: We first split $E [U ∣ s = 1; a_{1}]$ into the two cases $E [U ∣ (s, x) = (1, 0); a_{1}]$ and $E [U ∣ (s, x) = (1, 1); a_{1}]$ . The first of these equals $E [u ∣ (s, x) = (1, 0); a_{1}]$ by the definition of $U (\cdot, \cdot, \cdot)$ . For the second one, we use the linearity of the expectation to calculate $\begin{matrix} E [U ∣ (s, x) = (1, 1); a_{1}] & = E [v ∣ (s, x) = (1, 1); a_{1}] - E [v ∣ (s, x) = (1, 1); a_{1}] + E [u ∣ (s, x) = (1, 0); a_{1}] = E [u ∣ (s, x) = (1, 0); a_{1}] . \end{matrix}$ Hence, $E [U ∣ s = 1; a_{1}]$ equals $p (x = 0 ∣ s = 1; a_{1})$ times $E [u ∣ (s, x) = (1, 0); a_{1}]$ , plus $p (x = 1 ∣ s = 1; a_{1})$ times $E [u ∣ (s, x) = (1, 0); a_{1}]$ , again, directly implying that $E [U ∣ s = 1; a_{1}] = E [u ∣ (s, x) = (1, 0); a_{1}]$ .

Is this bad?

If the agent is able to influence $x$ as well as $s$ , then it can obviously set up an infinite improbability drive: It makes sure that the button definitely gets pressed (i.e., that $q (s = 1) = p (s = 1) = 1$ ), and it sets up a device that will cause $x$ to be true unless the agent wins the lottery. Then, $q (\cdot; a_{1}) = p (\cdot ∣ (s, x) = (1, 0); a_{1})$ ; i.e., according to the probability distribution the agent (effectively) uses to evaluate expected utility, it definitely wins the lottery.

So let’s now make the assumption that $x$ is independent of everything else: that is, that $p ((s, x, y); a_{1}) = p ((s^{'}, x, y^{'}); a_{1}^{'}) =: p (x)$ , for all $s, s^{'} \in S$ , all $x \in X$ , all $y, y^{'} \in Y$ , and all $a_{1}, a_{1}^{'} \in A_{1}$ . Under these assumptions, I can’t see a way to set up an infinite improbability drive (though it’s not immediately clear to me whether it’s provable that there is none).

However, this leaves the fact that the agent is acting as if it has a probability distribution that is drastically different from the correct one (since it acts as if $(s, x) = (1, 1)$ has probability zero, even though in fact the probability of $x$ is very high--- $1 - 10^{- 100}$ , say).

Since our agent is very smart, and it realizes that the probability distribution it cares about differs significantly from the probability distributions of the humans around it, it might come up with the bright idea of finding an enterprising businessperson, and proposing to them the following deal:

I will ensure that my shutdown button is pressed, i.e., I will make $s$ true. Under this assumption, you and I assign wildly different probability distributions to $x$ being true as well, so let’s make a bet: If $x$ is true, I will pay you all my current assets; if $x$ is false, you will pay me all of your assets (which is $10 \times$ as much). Since you think $x$ has only a one in $10^{100}$ chance of being false, you should think this is a great deal.

The businessperson agrees, and the agent loses all of its money.

I think this is still not quite Stuart’s thing, since my understanding is that he introduced $x$ in order to make sure that he’s never conditioning on a zero-probability event. The version above defines $U (\cdot, \cdot, \cdot)$ in a way that conditions on $s = 1$ , which could have probability $0$ according to certain actions.

This can be fixed by, e.g., changing the definition of $U (\cdot, \cdot, \cdot)$ to $U (a_{1}, (s, x, y), a_{2}) = ⎧ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎩ \begin{matrix} u (a_{1}, (s, x, y), a_{2}), & if s = x; v (a_{1}, (s, x, y), a_{2}) - E [v ∣ s = x; a_{1}] + E [u ∣ s \neq x; a_{1}], & else, \end{matrix}$ but this doesn’t fix the problem, it just makes the agent act as if it’s maximizing expected utility with respect to $p (\cdot ∣ s = x; a_{1})$ , which is still not the correct probability distribution, and still gives the agent an incentive to set up exactly the same deal as above.

Utility indifference and infinite improbability drives

Model

Infinite improbability drives

Stochastic events