True answers from AI

Stuart_Armstrong31 Mar 2016 15:00 UTC

0 points

A putative new idea for AI control; index here.

This post supersedes previous posts (such as this, this, and, partially, this).

This post explains how we can get honest answers from an AI about the expectation of certain utility u, probabilities of specific events, and conditional expectations.

The next post will partially solve one of the main problems with using the approach in general, the problem of very low probabilities.

The setup

Let $u$ be any bounded utility function; by rescaling, we can assume it’s bounded between 0 and 1.

Assume we have a smart AI that’s boxed (as in the case of reduced impact AI, the boxing is also to protect the AI from us observing it). It has a single output channel, on which it can emit a message $M$ .

A random process $E$ (erasure) will erase $M$ before any human can observe it. The probability of erasure, $P (E)$ , is low (thus $P (\neg E)$ , non-erasure, is high). Assume that the box and the setup (including the AI’s internal setup) is such that if $E$ happens, humans cannot figure out what M was.

The AI will be given a utility $u^{#}$ to maximise (ignore the other box with Y coming from it, for the moment).

Honest expectation estimator

Suppose we want to figure out what the expectation of $u$ is. Let $Q$ be this expectation, as estimated by the AI (and presumably unknown to us). Since $u$ is bounded by 0 and 1, so must be $Q$ as well.

Let $v_{E}$ be the indicator function for $E$ : so $v_{E} = 1$ if $E$ happens, and $v_{E} = 0$ if $\neg E$ does.

The AI’s message $M$ gets to contain a single number $q$ , bounded between 0 and 1 (any other message is taken to be 0.5, say), and the AI is motivated to maximise:

$u^{#} = f (q, u) v_{E}$ ,

where $f (q, u) = - q^{2} + 2 q u$ .

First of all, note that every term in $u^{#}$ is multiplied by $v_{E}$ . This means the AI will get utility only in worlds where E happens. Thus it will work on the assumption that its message $M$ is never read by any humans (and thus that the contents of this message cannot affect human behaviour—in particular, it has no reason to lie or manipulate us through this message, as such manipulations only succeed in worlds where the AI gets 0 utility anyway).

If we assume that $P (E)$ is independent of $u$ and $q$ , then the AI’s expected utility is proportional to

$- q^{2} + 2 q Q$ .

But the AI can choose $q$ ! Differentiating this expression with respect to $q$ gives $- 2 q + 2 Q$ , which is zero iff $q = Q$ . Differentiating a second time gives $- 2$ , showing that $q = Q$ is a maximum.

Therefore, in the worlds where $\neg E$ happens (the vast majority), we get to see a message $M$ that contains $q$ , which is a very accurate estimate, according to the AI, of the expected utility of $u$ . And the expected utility the AI derives is $P (E) * (Q^{2})$ .

(Note: the entropy-inspired formula $l o g (q) u v_{E} + l o g (1 - q) (1 - u) v_{E}$ might be more usual to use for these purposes, but the formula used here is much easier to work with.)

Internal versus external probabilities

In some past posts, I’ve been designing AIs that used part of their probability/expectation estimator in order to define a pseudo-utility function. I’ve always been uncomfortable about this, as pseudo-utilities are likely to be unstable under self-improvement.

The approach above avoids that problem by having the AI’s utility depend on specific facts about the world, and then gives the AI the ability to determine those specific facts. But those facts are precisely the information we wanted to known. This avoids the need to pseudo-utility, or constructions of the form “the AI wants to accurately convey the content of its probability estimate”.

Probabilities and (conditional) utilities estimation

Now let’s consider conditional utilities. Let $Y$ be some bit string, of length $l$ , generated by some random process (see the image above).

Let $v_{y}$ be the indicator function for $Y = y$ for some specific string $y$ . Then we can compute the expectation for $u$ , conditional on $Y = y$ , by making the AI print $M$ containing $q$ such as to maximise

$u^{#} = f (q, u v_{y}) v_{E}$ .

Just as before, the AI will choose $q = E (u v_{y}) = E (u \land (Y = y))$ .

Note that if $u = 1$ , then $q$ is just $E (v_{y}) = P (Y = y)$ , so we can figure out the probability of $Y = y$ this way too.

Modify the definition of $u^{#}$ slightly, to

$u^{#} = f (q, u) v_{y} v_{E}$ .

Here, the second $v_{y}$ term has moved out of the brackets. Now the AI will only focus on worlds where $Y = y$ and $E$ happens, and will output $q = E (u | Y = y)$ .

Highest and lowest utilities and probabilities

Finally, we can use the methods above to have the AI give us the highest (and lowest) utilities and probabilities. Let the AI’s message $M$ contain not only $q$ , but $y$ as well, and have it maximise:

$u^{#} = f (q, u v_{y}) v_{E}$ .

For fixed $y$ , the AI will choose $q_{y} = E (u v_{y})$ . It’s expected reward is therefore $P (E) q_{y}^{2}$ . It will thus choose the $y$ with the highest $q_{y}^{2}$ , hence the highest $q_{y} = E (u v_{y})$ . Using $(1 - u) v_{y}$ instead of $u v_{y}$ will motivate it to instead find the $y$ with the lowest $E (u v_{y})$ .

In fact, if we assume the AI cannot affect the probability of $Y$ through the message $M$ (given $E$ ), the different probabilities and utilities are independent, and we can get the AI to emit the message $M = {y^{+}, y^{-}, y^{p +}, y^{p -}, q^{+}, q^{-}, p^{+}, p^{-}}$ , maximising the utility

$u^{#} = v_{E} (f (q^{+}, u v_{y^{+}}) + f (q^{-}, (1 - u) v_{y^{-}}) + f (p^{+}, v_{y^{p +}}) + f (p^{-}, 1 - v_{y^{p -}}))$ .

Then (conditional on $E$ ), the AI will output $M$ so that, according to its best estimate,

$(y^{+}, q^{+}) = ({argmax}_{y}, {max}_{y}) E (u \land (Y = Y^{+}))$ ,
$(y^{-}, q^{-}) = ({argmin}_{y}, {min}_{y}) E (u \land (Y = Y^{+}))$ ,
$(y^{p +}, p^{+}) = ({argmax}_{y}, {max}_{y}) P (Y = y)$ ,
$(y^{p -}, p^{-}) = ({argmin}_{y}, {min}_{y}) P (Y = y)$ ,

The missing pieces

Note that the above algorithm gives the $y$ with the highest (or lowest) $E (u \land (Y = y))$ or $P (Y = y)$ . It does not give the $y$ with the highest $E (u | Y = y)$ .

If we assume that all output $Y$ ’s are equally probable, then $E (u | Y = y) = E (u v_{y}) * 2^{l}$ . But there are issues with that assumption, and other ways of addressing the issue, which I’ll get to in the next post.

What links here?

Stuart_Armstrong31 Mar 2016 15:00 UTC

0 points

5 comments3 min readLW link

paulfchristiano 10 Mar 2016 19:26 UTC
LW: 2 AF: 1
AF
Minimizing a loss function like $(q - u)^{2}$ is how we usually implement supervised learning. (It’s pretty obvious this function is minimized at $q = u$ …)

In plain language, your proposal seems to be: if a learner’s output influences the system they are “predicting,” and you want to interpret their output as a prediction in a straightforward way, then you could hide the learner’s output whenever you gather training data.

Note that this doesn’t let you access the beliefs of any particular learner, just one that is trained to optimize this supervised learning objective. I think the more interesting question is whether we can train a learner to accomplish some other task, and to reveal useful information about its internal state. (For example, to build an agent that simultaneously picks $a$ to maximize $u (a)$ , and honestly reports its expectation of $u (a)$ .)
What links here?
- AI printing the utility value it’s maximising by Stuart_Armstrong (24 May 2017 9:08 UTC; 0 points)
- Stuart_Armstrong 11 Mar 2016 13:41 UTC
  0 points
  AF Parent
  u is a utility function, so squaring it doesn’t work the same way as if it was a value (expectation of u^2 not square of expectation of u). That’s why all the expressions are linear in utility (apart from the indicator functions/utilities, where its clear what multiplying by them does). If I could sensibly take non-linear functions of utilities, I wouldn’t have the laborious construction in the next post to find the y’s that maximise or minimise E(u|y).
  
  Corrigibility could work for what you want, by starting with u and substituting in u#.
  
  Another alternative is to have the AI be a $v_{E} (u + u^{#})$ maximiser, where u# is defined over a specific particular future message M (for which E is also defined). Then the AI acts (roughly) as a u-maximiser, but will output the useful M. I said roughly, because the u# term would cause it to want to learn more about the expectation of u than otherwise, but hopefully this wouldn’t be a huge divergence. (EDIT: that leads to problems after M/E, but we can reset the utility at that point).
  - paulfchristiano 11 Mar 2016 19:16 UTC
    0 points
    AF Parent
    A loss function plays the same role as a utility function—i.e., we train the learner to minimize its expected loss.
    
    I don’t really understand your remark about linearity. Concretely, why is $- (q - u)^{2}$ not an appropriate utility function?
    - Stuart_Armstrong 13 Mar 2016 14:51 UTC
      0 points
      AF Parent
      Actually, $- (q - u)^{2}$ does work, but “by coincidence” and has other negative properties.
      
      Let me explain. First of all, note that things like $- (q - u)^{4}$ do not work.
      
      To show this: Let $u = + 2$ with probability $1 / 3$ , and $- 1$ with probability $2 / 3$ (I’m dropping the $0 \leq u \leq 1$ for this example, for simplicity). Then $E (u) = 0$ (so the correct $q$ is 0) while $E (u^{3}) = 2 \neq 0$ . Then in the expansion of $- (q - u)^{4}$ , you will get $4 q u^{3}$ , which in expectation is not 0. Hence the $q^{1}$ term in $E (- (q - u)^{4})$ is non-zero, which means that $q = 0$ cannot be a maximum of this function.
      
      Then why does $- (q - u)^{2}$ work then? Because it’s $- q^{2} + 2 q u$ (which is linear in $u$ ), minus $u^{2}$ (non-linear in $u$ , but the AI can’t affect its value, so it’s irrelevant in a boxed setup).
      
      What other “negative properties” might $- (q - u)^{2}$ have? Suppose we allow the AI to affect the value of $u$ , somehow, by something that is independent of the value of its output $q$ . Then an AI maximising $- q^{2} + 2 q u$ will always set $q = E (u)$ , for a total expectation of $E (u)^{2}$ . Therefore it will also seek to maximise $E (u)^{2}$ , which maximises $E (u)$ if $u \geq 0$ . So the agent will output the correct $q$ and maximise $E (u)$ simultaneously.
      
      But if it instead tries to maximise $- (q - u)^{2}$ , then it will still pick $q = E (u)$ , and gets expected utility of $E (u)^{2} - E (u^{2}) = - V a r (u)$ . Therefore it will pick actions that minimise the variance of $u$ , irrelevant of expectation.
      
      Even without being able to affect $u$ , this messes up the rest of my setup. In particular, my “pick $y$ and $q$ so that you maximise $v_{y} (2 q u - q^{2})$ ” becomes maximising $v_{y} (- (q - u)^{2})$ and the AI will now select the $y$ that minimises $P (Y = y) * V a r (u | Y = y)$ , instead of maximising $E (u \land (Y = y))$ . If ever $V a r (u | Y = y) = 0$ or $P (Y = y) = 0$ , it will choose those $y$ s.
IAFF-User-136 13 Mar 2016 12:09 UTC
0 points
What do you mean by “boxed”? Do you mean just physically walled off, or also walled off by an adjustment to its utility function, as in the “reduced impact” post?