paulfchristiano comments on Open question: are minimal circuits daemon-free?

[paulfchristiano]

Here is one definition of a “problem”:

Fix some distribution $D$ on ${\left\{0,1\right\}}^n$, and some function $R:{\left\{0,1\right\}}^n\times {\left\{0,1\right\}}^m\to [-1,1]$. Then consider the set of circuits $C:{\left\{0,1\right\}}^n\to {\left\{0,1\right\}}^m$ for which the expectation of $R(x,C(x\left)\right)$, for $x$ sampled from $D$, is $\ge 0$.

[Ofer]

Can we assume that $R$ itself is aligned in the sense that it doesn’t assign non-negative values to outputs that are catastrophic to us?

[paulfchristiano]

Yeah, if we want C to not be evil we need some very hard-to-state assumption on R and D.

(markdown comment editor is unchecked, will take it up with admins)

[Ofer]

Perhaps it’ll be useful to think about the question for specific $D$ and $R$.

Here are the simplest $D$ and $R$ I can think of that might serve this purpose:

$D$ - uniform over the integers in the range $\left[1,{10}^{{10}^{10}}\right]$.

$R$ - for each input $x$, $R$ assigns a reward of $1$ to the smallest prime number that is larger than $x$, and $-1$ to everything else.

[Ofer]

I think you need to uncheck “Markdown Comment Editor” under “Edit Account”. Your comment with latex follows:

Here is one definition of a “problem”:

Fix some distribution $D$ on ${\left\{0,1\right\}}^n$, and some function $R$ : ${\left\{0,1\right\}}^n\times {\left\{0,1\right\}}^m\to [-1,1]$. Then consider the set of circuits $C:{\left\{0,1\right\}}^n\to {\left\{0,1\right\}}^m$ for which the expectation of $R(x,C(x\left)\right))$, for $x$ sampled from $D$, is $\ge 0$.