Uninfluenceable learning agents

[Stuart_Armstrong]

A putative new idea for AI control; index here.

After explaining riggable learning processes, we can now define influenceable (and uninfluenceable) learning processes.

Recall that the (unriggable) influence problem is due to agents randomising their preferences, as a sort of artificial `learning’ process, if the real learning process is slow or incomplete.

---

Suppose we had a learning process that it wasn’t possible to influence. What would that resemble? It seems like it must be something where the outcome of the learning process depends only upon so outside fact about the universe, a fact the agent has no control over.

So with that in mind, define:

Definition: A learning process $P$ on the POMDP $\mu$ is initial-state determined if there exists a function $f_P:S\to \Delta R$ such that $P$ factors through knowledge of the initial state $s_0$. In other words:

• $P(\cdot \mid h_m)={\sum }_{s\in S}\mu (s_0=s\mid h_m)f_P\left(s\right).$

Thus uncertainty about the correct reward function comes only from uncertainty about the initial state $s_0$.

This is a partial definition, but an incomplete one. To finalise it, we need the concept of counterfactually equivalent POMDPs:

Definition: A learning process $P$ on $\mu$ is uninfluenceable if there exists a counterfactually equivalent ${\mu }^{\prime }$ such that $P$ is initial-state determined on ${\mu }^{\prime }$.

Though the definition of unriggable and uninfluenceable seem quite different, they’re actually quite closely related, as we’ll see in a subsequent post. Uninfluenceable can be seen as `unriggable in all background info about the universe’. In old notation terms, rigging is explored in the sophisticated cake or death problem, (unibased) influence in the ultra-sophisticated version.

Example

Consider the environment $\mu$ presented here:

In this POMDP (actually MDP, since it’s fully observed), the agent can wait for a human to confirm the correct reward function (action $a^w$) or randomise its reward (action $a^r$). After either actions, the agent gets equally likely feedback $0$ or $1$ (states $s^{wi}$ and $s^{ri}$, $0\le i\le 1$).

We have two plausible learning processes: $P$, where the agent learns only from the human input, and $P^{\prime }$, where the agent learns from either action. Technically:

• $P(R_i\mid o_0{a}^w{o}^{wi})=P^{\prime }(R_i\mid o_0{a}^w{o}^{wi})=1$,

• $P(R_i\mid o_0{a}^w{o}^{wj})=0.5$ for all $0\le i,j\le 1$,

• $P^{\prime }(R_i\mid o_0{a}^w{o}^{wi})=1$,

with all other probabilities zero.

Now, $\mu$ is counterfactually equivalent to ${\mu }^{\prime \prime }$:

And on ${\mu }^{\prime \prime }$, $P$ is clearly initial-state determined (with $f_P\left(s_0^i\right)\left(R_i\right)=1$), and is thus uninfluenceable on ${\mu }^{\prime \prime }$ and $\mu$.

On the other hand, $P^{\prime }$ is initial-state determined on ${\mu }^{\prime }$:

However, ${\mu }^{\prime }$ is not counterfactually equivalent to $\mu$. In fact, there is no PORMDP counterfactually equivalent to $\mu$ on which $P^{\prime }$ is initial-state determined, so $P^{\prime }$ is not uninfluenceable.