Stuart_Armstrong comments on Double indifference is better indifference

Stuart_Armstrong 6 May 2016 15:32 UTC
LW: 1 AF: 1
AF
This is my punishment for being sloppy with definitions :-(

Ok, let’s be clearer. When the button is pressed, a process begins, to change $u$ into $v$ . But the process is stocastic, and has a small chance of failing. The failure of this is designated by $u \to u$ .

So if Pr refers to pressing the button, the expected reward is:

$P (\neg P r) E (u | \neg P r) + P (P r, u \to u) E (u | P r, u \to u) + P (P r, u \to v) (E (v | P r, u \to v) + E (u | P r, u \to u) - E (v | P r, u \to v)) = P (\neg P r) E (u | \neg P r) + (P (P r, u \to u) + P (P r, u \to v)) E (u | P r, u \to u) = (1 / 2) 3 + (1 / 2) 8 = 5.5.$
- jessicata 6 May 2016 19:33 UTC
  LW: 2 AF: 1
  AF Parent
  This makes more sense, thanks for explaining!
  
  I’m going to translate this to my notation so I understand it better (feel free to skip this, it’s mostly for my benefit). Let $X$ be the event that the coin flip succeeds (which usually happens). We can say that $X$ still has a value even if $\neg P r$ (its value just doesn’t do anything). So $u \to v$ happens iff $P r \land X$ .
  
  The expected utility is then
  
  $E [[\neg P r \lor \neg X] u + [P r \land X] (v + E [u | P r \land \neg X] - E [v | P r \land X])$
  
  Suppose the AI never updates its beliefs (because it goes into stasis the first round). Then this is
  
  $E [[\neg P r \lor \neg X] u] + E [[P r \land X] v] + P (P r \land X) E [u | P r \land \neg X] - E [[P r \land X] v]$ $= E [[\neg P r \lor \neg X] u] + P (P r \land X) E [u | P r \land \neg X]$ $= P (\neg P r) E [u | \neg P r, \neg X] + P (P r \land X) E [u | P r, \neg X]$ $\approx P (\neg P r) E [u | \neg P r, \neg X] + P (P r) E [u | P r, \neg X]$ $= E [u | \neg X]$ $= E [u | d o (\neg X)]$ (I wrote it as a causal counterfactual in the last step; it doesn’t make a difference since $X$ has no causal ancestors, but I find it easier to reason about this way)
  
  So in the end, if the whole game is to construct a subagent and then go into stasis, then the agent is just a $u$ -maximizer who believes $\neg X$ (as you said). It avoids the gerrymandering problem by doing a causal counterfactual.
  
  I think another way of framing this is that the button always succeeds, and agent optimizes $E [u | d o (\neg P r)]$ (which relates this back to the causal version of factoring out effects, where you optimize $q (P r) E [v | d o (P r)] + q (\neg P r) E [u | d o (\neg P r)]$ ; you get utility indifference with stasis by setting $q (P r) = 0$ ). This will be slightly different in that here the agent “believes” it will be magically prevented from pressing the button, whereas originally the agent just “believes” that the button will fail.