Comparing reward learning/​reward tampering formalisms

[Stuart_Armstrong]

Contrasting formalisms

Here I’ll contrast the approach we’re using in using in Pitfalls of Learning a Reward Online (summarised here), with that used by Tom Everitt and Marcu Hutter in the conceptually similar Reward Tampering Problems and Solutions in Reinforcement Learning. In the following, histories $h_i$ are sequences of actions $a$ and observations $o$; thus $h_i=a_1{o}_1{a}_2{o}_2\dots a_i{o}_i$. The agent’s policy is given by $\pi$, the environment is given by $\mu$.

Then the causal graph for the “Pitfalls” approach is, in plate notation (which basically means that, for every value of $j$ from $1$ to $n$, the graph inside the rectangle is true):

The $R$ is the set of reward functions (mapping “complete” histories $h_n$ of length $n$ to real numbers), the $\rho$ tells you which reward is correct, conditional on complete histories, and $r$ is the final reward.

In order to move to the reward tampering formalism, we’ll have to generalise the $R$ and $\rho$, just a bit. We’ll allow $R$ to take partial histories - $h_j$ shorter than $h_n$ - and return a reward. Similarly, we’ll generalise $\rho$ to a conditional distribution on $R$, conditional on all histories $h_j$, not just on complete histories.

This leads to the following graph:

This graph is now general enough to include reward tampering formalism.

States, data, and actions

In reward tampering formalism, “observations” ($o_j$) decompose into two pieces: states ($S_j$) and data ($D_j$). The idea is that data informs you about the reward function, while states get put into the reward function to get the actual reward.

So we can model this as this causal graph (adapted from graph 10b, page 22; this is a slight generalisation, as I haven’t assumed Markovian conditions):

Inside the rectangle, the histories split into data ($D_{1:j}$), states ($S_{1:j}$), and actions ($a_{1:j}$). The reward function is defined by the data only, while the reward comes from this reward function and from the states only—actions don’t directly affect these (though they can indirectly affect them by deciding what states and data come up, of course). Note that in the reward tampering paper, the authors don’t distinguish explicitly between $R_j$ and $r_j$, but they seem to do so implicitly.

Finally, ${\Theta }_{\ast }^R$ is the “user’s reward function”, which the agent is estimating via $D_{1:j}$; this connects to the data only.

Almost all of the probability distributions at each node are “natural” ones that are easy to understand. For example, there are arrows into $r_j$ (the reward) from $R_j$ (the reward function) and $S_{1:j}$ (the states history); the “conditional distribution” of $r_j$ is just “apply $R_j$ to $S_{1:j}$. The environment, action, and history naturally provide the next observations (state and data).

Two arrows point to more complicated relations: the arrow from ${\Theta }_{\ast }^R$ to $D_j$, and that from $D_{1:j}$ to $R$. The two are related; the data $D_j$ is supposed to tell us about the user’s true reward function, while this information informs the choice of $R$.

But the fact that the nodes and the probability distribution have been “designed” this way doesn’t affect the agent. It has a fixed process $P_{rt}(R\mid D_{1:j})$ for estimating $R$ from $D_{1:j}$ ($P_{rt}$ stands for the probability function for the reward tampering formalism). It has access to $a_j$, $D_j$, and $S_j$ (and their histories) as well as its own policy, but has no direct access to $\mu$ or ${\Theta }_{\ast }^R$.

In fact, from the agent’s perspective, ${\Theta }_{\ast }^R$ is essentially part of $\mu$, the environment, though focusing on the $D_j$ only.

States and actions in “Pitfalls” formalism

Now, can we put this into the “Pitfalls” formalism? It seems we can, as so:

All conditional probability distributions in this graph are natural.

This graph look very similar to the “reward tampering” one, with the exception of ${\rho }_j$ and ${\Theta }_{\ast }^R$, pointing at $R_j$ and $D_j$ respectively.

In fact, ${\rho }_j$ play the role of $P_{rt}(R\mid D_{1:j})$ in that, for $P_{lp}$ the probability distribution for learning process,

$$P_{lp}(R\mid D_{1:j},{\rho }_j)=P_{rt}(R\mid D_{1:j}).$$

Note that $P_{lp}$ in that expression is natural and simple, while $P_{rt}$ is complex; essentially $P_{rt}$ carries the same information as ${\rho }_j$.

The environment ${\mu }_{lp}$ of the learning process plays the same role as the combined ${\mu }_{rt}$ and ${\Theta }_R^{\ast }$ from the reward tampering formalism.

So the isomorphism between the two approaches is, informally speaking:

1. On reward functions conditional on histories, $P_{rt}\leftrightarrow \rho$.

2. ${\mu }_{lp}\leftrightarrow ({\mu }_{rt},{\Theta }_R^{\ast })$.

Uninfluenceable similarities

If we make the processes uninfluenceable (a concept that exists for both formalisms), the causal graphs look even more similar:

Here the pair $({\mu }_{lp},\eta )$, for the learning process, play exactly the same role as the pair^[1] $({\mu }_{rt},{\Theta }_{\ast }^R)$, for reward tampering: determining reward functions and observations.

---

1. ↩︎

   There is an equivalence between the pairs, but not between the individual elements; thus ${\mu }_{lp}$ carries more information than ${\mu }_{rt}$, while $\eta$ carries less information than ${\Theta }_{\ast }^R$.

[RyanCarey]

It would be nice to draw out this distinction in more detail. One guess:

• Uninfluencability seems similar to requiring zero individual treatment effect of D on R.

• Riggability (from the paper) would then correspond to zero average treatment effect of D on R

[algon33]

Stuart, by ”$P_{rt}$$\left(R|D_{1;j}\right)$ is complex” are you referring to their using $R=R(.,E[{\Theta }_{\ast }^R|D_{1;j}\left]\right)$ as the estimated reward function?

Also, what did you think of their arguement that their agents have no incentive to manipulate their beliefs because they evaluate future trajectories based of their current beliefs about how likely they are? Does that suffice to implement eq. 1) from your motivated value selection paper?

[Stuart_Armstrong]

Suart, by ”$P_{rt}(R\mid D_{1:j})$ is complex” are you referring to...

I mean that that defining $P_{rt}$ can be done in many different ways, and hence has a lot of contingent structure. In contrast, in $P_{lp}(R\mid D_{1:j},\rho )$, the $\rho is a complex distribution on $R$, conditional on $D_{1:j}$; hence $P_{lp}$ itself is trivial and just encodes “apply $\rho$ to $R$ and $D_{1:j}$ in the obvious way.