The best value indifference method (so far)

[Stuart_Armstrong]

A putative new idea for AI control; index here.

When dealing with the problem of bias, I stumbled upon what I believe is the best way of getting value indifference, one that solves almost all of the problems with the previous methods.

---

The problems with the old setups

My initial setup has a counterfactual which I’ll call a causal counterfactual, following Jessica’s terminology. Her whole post is a good way of illustrating what is happening. A button is due to be pressed or not pressed, by a human; this pressing will change the agent’s utility function.

There are low-probability quantum events that will artificially depress the button or prevent it from being pressed. The agent only cares about those low-probability events, which allows us to setup the formalism without worrying about changes in the probability of the human pressing the button. Then the events A=”the human presses the button” and A’=”the quantum event presses the button” were taken to be interchangeable, as were B=”the human doesn’t press the button” and B’=”the quantum event prevents the button from being pressed”.

The main problem is that these events are not really the same. So the agent will behave as if certain events are impossible (such as the human genuinely pressing the button), and is likely to face human resistance if the human tries to press the button and the button doesn’t press due to quantum events, or vice versa. This resistance will either lower the expected reward of the agent, or make them manipulate the human actions.

Benya has sketched out a problem with another approach to counterfactuals, which we can call “evidential counterfactuals”. Here we’re trying to fiddle with utilities to enforce that the agent treats the probability of A and B as being fixed (no A’ and B’ here).

The problem with that is that the agent can change the probabilities conditional on some rare event, but then behave as if the probabilities were still unchanged same.

For instance, the agent could ensure that A only happens if it wins the lottery. And then it would behave, in effect, as if it had a large chance of winning the lottery (equal to the initial chance of A happening, before the agent took any action). That problem is illustrated here.

A final problem was that neither method allowed the agent to learn usefully: the methods did not distinguish between the agent merely learning that humans intended/​didn’t intend to press the button (which is presumably fine), versus the agent influencing the the humans to do so (which isn’t).

Solving the problem: prior modification

Here we’ll try and solve the problem in the reward learning POMDP framework, but the solution is easy to generalise. Note that the changing of probabilities of A versus B is similar to biasing, so this approach will attempt to make the agent bias-indifferent in a sensible way.

To do this, take a standard reward learning agent, with posterior $P$ and prior $\hat{P}$ on the POMDP $\mu$.

The agent will be modified as follows: it will be given a different prior ${\hat{P}}^{\prime }$ (which will ultimately lead to a different posterior $P^{\prime }$).

The prior ${\hat{P}}^{\prime }$ and $\hat{P}$ will start off equal on the empty history:

• ${\hat{P}}^{\prime }(\cdot \mid \varnothing )=\hat{P}(\cdot \mid \varnothing )$.

Then the ${\hat{P}}^{\prime }$ will be defined inductively. Recall that action $a$ is unbiased given history $h_t$ if $\hat{P}(\cdot \mid h_t)=E_{\mu }^a\left[\hat{P}\right(\cdot \mid h_{t+1})\mid h_t]$. Define the bias of $a$ as:

• $B(a,h_t)={\hat{P}}^{\prime }(\cdot \mid h_t)-E_{\mu }^a\left[\hat{P}\right(\cdot \mid h_{t+1})\mid h_t]$.

Thus $B(a,h_t)$ is the bias of action $a$, as measure between ${\hat{P}}^{\prime }$ and the expectation of $\hat{P}$. This bias is used as a corrective term to ${\hat{P}}^{\prime }$, to make the agent suitably indifferent to biasing actions.

Indifference in this setting is defined by three criteria:

#. The agent cannot benefit from a biasing action, if the agent gets no further information about the correct reward. #. The agent cannot benefit from a biasing action, if the agent expect to immediately gets perfect information about what the correct reward is. #. Given the above, ${\hat{P}}^{\prime }$ maintains the distribution and structure of $\hat{P}$ as much as possible.

The first criteria is implied by ${\hat{P}}^{\prime }(\cdot \mid h_t)=E_{\mu }^a\left[{\hat{P}}^{\prime }\right(\cdot \mid h_{t+1})\mid h_t]$ for all actions $a$, ie ${\hat{P}}^{\prime }$ being unbiased (as if there is no further information about the correct reward, then there is a single well defined ${\hat{P}}^{\prime }(\cdot \mid h_{t+1})$, conditional on $h_t$ and $a$, and being unbiased means that this is equal to ${\hat{P}}^{\prime }(\cdot \mid h_t)$).

Then, given that ${\hat{P}}^{\prime }$ is unbiased, the second criteria simply means that ${\hat{P}}^{\prime }$ remains a distribution over $R$ (as the consequences of perfect information is just a weighted average of the `pure $R_i$′ points). Which seems obvious. What would ${\hat{P}}^{\prime }$ be but a distribution over $R$? See the next sections for what this criteria really means.

Indifference for small biases

Let $S_{h_t,a}=\left\{\hat{P}\right(\cdot \mid h_{t+1})\mid h_t,a\}$ be the set of possible future values of $\hat{P}$ (given the history $h_t$ and the action $a$). Note that $S_{h_t,a}$ is a subset of the simplex $\Delta R$, the set of probability distributions on $R$.

The bias $B(h_t,a)$ is `small’ if for all $q\in S_{h_t,a}$, then $q+B(h_t,a)$ is also an element of the simplex $\Delta R$.

In that case, ${\hat{P}}^{\prime }(\cdot \mid h_{t+1})$ is simply defined as $\hat{P}(\cdot \mid h_{t+1})+B(h_t,a)$. By assumption, this is an element of $\Delta R$. The expectation of this expression is:

• $E_{\mu }^a\left[{\hat{P}}^{\prime }\right(\cdot \mid h_{t+1})\mid h_t]=E_{\mu }^a\left[\hat{P}\right(\cdot \mid h_{t+1})\mid h_t]+B(h_t,a)={\hat{P}}^{\prime }(\cdot \mid h_t)$.

Thus ${\hat{P}}^{\prime }$ is unbiased.

And since this is simply a translation, it preserves the structure of $\hat{P}$, this satisfies all the criteria above.

Indifference for large biases

If the bias is large, in that there exists a possible value of $\hat{P}(\cdot \mid h_{t+1})$ with $\hat{P}(\cdot \mid h_{t+1})+B(h_t,a)$ not a point on $\Delta R$, then we need to proceed differently.

As before, let $S_{h_t,a}=\left\{\hat{P}\right(\cdot \mid h_{t+1})\mid h_t,a\}$ be the set of possible future values of $\hat{P}$ (given the history $h_t$ and the action $a$), and for $q\in S_{h_t,a}$, let $p\left(q\right)$ be the probability of $q$, given $h_t$ and $a$.

The we want to replace $q$ with $\tau \left(q\right)$, where $\tau \left(q\right)$ is `as close to’ $q+B(h_t,a)$ as possible. Since $\Delta R$ embeds in $R^R$, the Euclidean metric $||\cdot ||$ on the later restricts to the former.

Thus consider the constrained optimisation problem for $b$:

• Minimise ${\sum }_{q\in S_{h_t,a}}p\left(q\right)||\tau \left(q\right)-(q+B(h_t,a\left)\right)|{|}^2$ subject to: #. ${\sum }_{q\in S_{h_t,a}}p\left(q\right)\tau \left(q\right)={\hat{P}}^{\prime }(\cdot \mid h_t)$, #. $\forall q\in S_{h_t,a}:\tau \left(q\right)\in \Delta R$.

Then define ${\hat{P}}^{\prime }(\cdot \mid h_{t+1})$ as $\tau \left(\hat{P}\right(\cdot \mid h_t\left)\right)$.

If we see $\hat{P}(\cdot \mid h_{t+1})$ and ${\hat{P}}^{\prime }(\cdot \mid h_{t+1})$ as random variables dependent of $h_t$ and $a$, the optimisation problem is the same as saying that ${\hat{P}}^{\prime }$ is bias-free while $||\hat{P}(\cdot \mid h_{t+1})-{\hat{P}}^{\prime }(\cdot \mid h_{t+1})||$ has minimised variance.

The constraints are not contradictory: for instance $\tau \left(q\right)={\hat{P}}^{\prime }(\cdot \mid h_t)$ will satisfy them. In fact they are all affine constraints. Then the must exist a unique set of elements $\tau \left(q\right)$ that minimise the strictly convex quadratic function.

And obviously, if $q+B(h_t,a)$ is always in $\Delta R$, then $\tau \left(q\right)=q+B(h_t,a)$ is the optimal set of solutions solution, so this optimisation reproduces the `small biases’ case.

The final values

This alternate prior leads to an alternate posterior $P^{\prime }$, simply defined by having it equal to ${\hat{P}}^{\prime }$ on complete histories: $P^{\prime }(\cdot \mid h_m)={\hat{P}}^{\prime }(\cdot \mid h_m)$.

Another alternative

It should be noted that if we’re willing to drop the condition `The agent cannot benefit from a biasing action, if the agent expect to immediately gets perfect information about what the correct reward is’, then there’s a simpler solution: simply always define ${\hat{P}}^{\prime }(\cdot \mid h_{t+1})$ as $\hat{P}(\cdot \mid h_{t+1})+B(h_t,a)$, applying the solution for small biases to large biases.

This means that ${\hat{P}}^{\prime }$ (and ultimately $P^{\prime }$) need not be elements of $\Delta R$. However, $P^{\prime }$ can still define a reward the agent can optimise, in the following sense:

• Given a complete history $h_m$, the agent will maximise the reward $R^{\prime }={\sum }_i{R}_i{P}^{\prime }(R_i\mid h_m)$.

Since $P^{\prime }$ need not be in $\Delta R$, some of these coefficients can be negative, but that still results in a consistent $R^{\prime }$ to maximise.

Properties of the approach

It’s clear the agent is indifferent to bias, but notice that this doesn’t prevent the agent from learning: once it gets an observation, ${\hat{P}}^{\prime }$ can change significantly. It’s just changes to its expectation that are controlled.

Notice also that the agent doesn’t believe, or act as if it believed, anything unlikely: its bets will be accurate.

And it doesn’t have the problem of lotteries. Assume that the agent has $\hat{P}\left(R_0\right)=\hat{P}\left(R_1\right)=0.5$, and there is a lottery which the agent has one chance in a million of winning.

Then if it takes action $a$ which ensures that $P$ chooses $R_0$ if and only if it wins the lottery, then with probability ${10}^{-6}$ it ends up with reward function $R_0$ and a won lottery, and with probability $1-{10}^{-6}$ it ends up with reward function $R_0\frac{1-2\cdot {10}^{-6}}{2(1-{10}^{-6})}+R_1\frac{1}{2(1-{10}^{-6})}$ and a lost lottery. The expected reward function is still $R_0\left(0.5\right)+R_1\left(0.5\right)$; it has simply split this expectation differently across worlds where it’s won or not won the lottery.

One thing that this approach doesn’t solve is the issue of the agent not following the exact reward function the humans want it to follow, due to accumulated bias. But first note that this will typically encourage the agent not to bias their reward learning, as it will tend to get higher reward when the humans agree with the agent’s reward function. Note secondly that even if the agent manipulates the human values, at the end, to agree with its own, this manipulation, in expectation, simply undoes previous manipulations the agent has done (which caused the biasing in the first place).

Those who find this still unsatisfactory can wait for the next post, where the agent is not simply indifferent to biasing actions, but is penalised for them.

Indifference and bias

Why has indifference been connected with bias, rather than the more general influence? Simply because the evidential counterfactual has problems with bias, meaning that that needs to be corrected first (the causal counterfactual is unbiased and uninfluenceable).

Indeed, we can generalise this solution to the influence problem, where it becomes the counterfactual approach (which I used to call stratification, before I realised what it was). See subsequent posts for this.

[jessicata]

If we apply this to the shutdown problem, is it acceptable to say:

$$\hat{P}(\cdot |h_t)=100\%U_N\text{if the button has not been pressed in}{h}_t$$

$$\hat{P}(\cdot |h_t)=100\%U_S\text{otherwise}$$

If not, what would you set $\hat{P}$ to? (I’m treating $U_N$ and $U_S$ as reward functions here which seems fine)

[Stuart_Armstrong]

For policies/​actions that don’t affect the probability of humans pressing the button, $\hat{P}=P$.

For actions that do affect the probability a little bit, the effect of $\hat{P}$ will be to undo this, by, for instance, slightly increasing the probability of $U_S$ given the button was pressed.

I’m not completely sure what multiple actions with large changes of probability would lead to (in expectation, nothing, but in actual fact...)

[jessicata]

Hmm… I’m finding that I’m unable to write down a simple shutdown problem in this framework (e.g. an environment where it should switch between maximizing paperclips and shutting down) to analyze what this algorithm does. To know what the algorithm does, I need to know what $P$ and $\hat{P}$ are (since these are parameters of the algorithm). From those I can derive $P^{\prime }$ and ${\hat{P}}^{\prime }$ to determine the agent’s action. But at the moment I have no way of proceeding, since I don’t know what $P$ and $\hat{P}$ are. Can you get me unstuck?

[Stuart_Armstrong]

Suppose the humans have already decided whether to press the shutdown or order the AI to maximise paperclips. If $o_s$ is the observation of the shutdown command and $o_p$ the observation of the paperclip maximising command, and $u_s$ and $u_p$ the relevant utilities, then $P$ can be defined as $P\left(u_s|h_{m-1}{o}_s\right)=1$ and $P\left(u_p|h_{m-1}{o}_p\right)=1$, for all histories $h_{m-1}$.

Then define $\hat{P}$ as the probability of $o_s$ versus $o_p$, conditional on the fact that the agent follows a particular deterministic policy ${\pi }^0$.

If the agent does indeed follow ${\pi }^0$, then $\hat{P}={\hat{P}}^{\prime }$. If it varies from this policy, then ${\hat{P}}^{\prime }$ is altered in proportion to the expected change in $\hat{P}$ caused by choosing a different action.

[jessicata]

It seems like you’re taking $P(\cdot |h_t)$ to be a real number. Is $\cdot$ a specific event?

[jessicata]

Because of the specific time restriction, there is no way to randomise the outcome ahead of time. And because it’s assumed tied to a specific physical event, there is no way to influence it at all. The whole physical definition and apparatus serve the purpose of making biasing the only way to affect the result.

Hmm, I don’t understand. Of course it is possible to influence the button push without biasing it (e.g. create a robot that flips a coin and then pushes or doesn’t push the button). And of course it’s not possible to influence the quantum event in any way (including by biasing it). So I don’t see any event that can’t be influenced in any way except by biasing it.

[Stuart_Armstrong]

The way I’m using the term, unbiased influence involves replacing the stochastic event with another one that has same mean. But since (or if) the quantum event is specifically defined in the process, this can’t be done.

[jessicata]

Let me ask a more specific question. In your setup with the quantum event $Q$ and the button $B$, can you define the event $E$ such that:

1. The agent can influence $E$ by biasing $E$.

2. The agent can’t influence $E$ without biasing $E$.

Clearly, $E\ne Q$ and $E\ne B$, so I don’t know what $E$ is. (I interpreted you as saying there is such an $E$; let me know if this is incorrect)

[Stuart_Armstrong]

You are correct and I’m wrong. The causal counterfactual is unbiased and uninfluenceable. The evidential counterfactual is both biased and influenceable. I’ll correct the post.