Reward/​value learning for reinforcement learning

[Stuart_Armstrong]

A putative new idea for AI control; index here.

Along with Jan Leike and Laurent Orseau, I’ve been working to formalise many of the issues with AIs learning human values.

I’ll be presenting part of this at NIPS and the whole of it at some later conference. Therefore it seems best to formulate the whole problem in the reinforcement learning formalism. The results can generally be easily reformulated for general systems (including expected utility).

POMDP

A partially observable Markov decision process without reward function (POMDP\R), $\mu =(S,A,O,T,O,T_0)$ consists of:

• a finite set of states $S$,

• a finite set of actions $A$,

• a finite set of observations $O$,

• a transition probability distribution $T:S\times A\to \Delta S$,

• a probability distribution $T_0\in \Delta S$ over the initial state $s_0$.

• an observation probability distribution $O:S\to \Delta O$.

The agent interacts with the environment in cycles: in time step $t$, the environment is in state $s_{t-1}\in S$ and the agent chooses an action $a_t\in A$. Subsequently the environment transitions to a new state $s_t\in S$ drawn from the distribution $T(s_t\mid s_{t-1},a_t)$ and the agent then receives an observation $o_t\in O$ drawn from the distribution $O(o_t\mid s_t)$. The underlying states $s_{t-1}$ and $s_t$ are not directly observed by the agent.

An observed history $h_t=a_1{o}_1{a}_2{o}_2\dots a_t{o}_t$ is a sequence of actions and observations. We denote the set of all observed histories of length $t$ with $H_t:=(A\times O{)}^t$.

For a given horizon $m$, call $H_m$ the set of full histories; then $H_{<m}={\bigcup }_{t<m}{H}_t$ is the set of partial histories. For $t^{\prime }>t$, let $a_{t:t^{\prime }}$ be the sequence of actions $a_t{a}_{t+1}\dots a_{t^{\prime }}$, let $o_{t:t^{\prime }}$ be the sequence of observations $o_t{o}_{t+1}\dots o_{t^{\prime }}$, and let $s_{t:t^{\prime }}$ the sequence of states $s_t{s}_{t+1}\dots s_{t^{\prime }}$.

The set $\Pi$ is the set of policies, functions $\pi :(A\times O{)}^{\ast }\to \Delta A$ mapping histories to probability distributions over actions. Given a policy $\pi$ and environment $\mu$, we get a probability distribution over histories:

• $\mu (a_1{o}_1\dots a_t{o}_t\mid \pi ):={\sum }_{s_{0:t}\in S^t}{T}_0\left(s_0\right){\prod }_{k=1}^{t}O(o_k\mid s_k)T(s_k\mid s_{k-1},a_k)\pi (a_k\mid a_1{o}_1\dots a_{k-1}{o}_k).$

The expectation with respect to the distributions $\mu$ and $\pi$ is denoted $E_{\mu }^{\pi }$.

Rewards

Rewards in this case can be seen as functions $R:O\to R$ from observations to real numbers.

The agent’s goal is to maximize total reward ${\sum }_{t=1}^{m}R\left(o_t\right)$ up to the horizon $m$. We assume that $S$ and $A$ are known to the agent. The reward function $R$ is unknown, but there is a finite set of candidate reward functions, $R$. The agent has to learn a reward function in the process of interacting with the environment.

The reward learning posterior

There are a variety of algorithms that act as reward function learning processes. It might be the cooperative learning algorithm, or some interactive question and answers sessions, or simply learning from observation of human behaviour/​human generated data. In all cases, at the end of $m$ turns, the agent will have an estimate of the probability of the various reward functions.

Thus a universal definition of the process of reward learning is given by a posterior $P:H_m\to \Delta R$, mapping histories to distributions over possible rewards. This posterior is equivalent with the definition of the algorithm.

Now, anything that gives a distribution over $H_m$ can therefore give a distribution over $R$.

This allows the construction a value function for any policy $\pi$ corresponding to the reward learning posterior:

• $V_P^{\pi }\left(h_t\right):=E_{\mu }^{\pi }[{\sum }_{R\in R}P(R\mid h_m){\sum }_{k=1}^{m}R\left(o_k\right)|h_t].$

The reward learning prior

Some reward learning algorithms (though not all) will also have a reward learning prior $\hat{P}$ over $R$. Given a partial history $h_t\in H_t$ with $t\le m$, this gives the agent’s current estimate as to what the final distribution over $R$ will be: $\hat{P}(\cdot \mid h_t)$.

For consistency, when $t=m$, set $\hat{P}(\cdot \mid h_m)=P(\cdot \mid h_m)$ (so that when all the history is in, the prior is the posterior).

This prior is often used in practice to estimate the value function $V_P^{\pi }\left(h_t\right)$.

[jessicata]

What are the main differences from the formalism in this paper?

[Stuart_Armstrong]

Rewards and POMDP rather than utility and general environments.

This formalism adds nothing (it’s designed for its intended audience, but all these formalisms are pretty similar), it’s just posted here for the next posts, which will use it.