Donald Hobson comments on The easy goal inference problem is still hard

[Donald Hobson]

Let $W$be the set of Worlds, $U:W\to R$ be the set of all utility functions , and $O$ the set of human observations, and $A$ the set of human actions Let $C=\{U\times O\to A\}$ be the set of bounded optimization algorithms, so that an individual $c\in C$ is a function from (Utility, Observation) pairs, to actions. Examples of $c$ include AIXI-tl with specific time and length limits, and existing deep RL models. This consists of the AI’s idea about what kind of bounded agent we might be. There are various conditions of approximate correctness on $C$

Let $O^{\ast }$ and $A^{\ast }$be the AI’s observation and action space

The AI is only interacting with one human, and has a prior $\Pi =O\times A\times C\times U\times W\times O^{\ast }\times A^{\ast }\to R$ where $W$ stands for the rest of the world. Note that parameters not given are summed over, $\Pi (o,c,u,w,o^{\ast },a^{\ast })={\sum }_{a\in A}\Pi (o,a,c,u,w,o^{\ast },a^{\ast })$

The AI performs Bayesian updates on $\Pi$ as normal. Gathering part of an observation $o^{\prime }$

$${\Pi }_{\text{new}}\propto \{\begin{array}{cc}\Pi & o^{\ast }\Rightarrow o^{\prime }\\ 0& \text{else}\end{array}$$

If $A^{\ast }$ is the AI’s action space, it chooses ${\text{argmax}}_{a^{\ast }\in A^{\ast }}\left({\sum }_{w\in W}\right(E_{\Pi }\left(u\right(w\left)\right)\times P\left(w\right))$

$$\underset{a^{\ast }\in A^{\ast }}{argmax}\left(\sum _{w\in W}\left(E_{\Pi }\right(u\left(w\right))\times P(w\left)\right)\left)\right)\right)=\underset{a^{\ast }\in A^{\ast }}{argmax}\left(\sum _{u\in U,w\in W}\left(\Pi \right(u,w,a^{\ast })\times u(w\left)\right)\left)\right)\right)$$

Of course, a lot of the magic here is happening in $\Pi$, bit if you can find a prior that favours fast and approximately correct optimization algorithms $C$ over slow or totally defective ones and favours Simplicity of each terms.

Basically the humans utility function is

$$u_{\text{h}}\left(w\right)=\sum _{o\in O,c\in C,u\in U}\Pi (o,a(o),c,u)\times u\left(w\right)$$

Where $O$ is the set of all things the human could have seen, $a\left(o\right)$ is whatever policy the human implements, and $\Pi$ focuses on $c\in C$ that are simple, stocastic, bounded maximization algorithms.

If you don’t find it very clear what I’m doing, thats ok. I’m not very cleasr what I’m doing. This is a bit of a point in the rough direction.

[paulfchristiano]

A lot of magic is happening in the prior over utility functions and optimization algorithms, removing that magic is the open problem.

(I’m pessimistic about making progress on that problem, and instead try to define value by using the human policy to guide a process of deliberation rather than trying to infer some underlying latent structure.)