jessicata comments on Double indifference is better indifference

jessicata 5 May 2016 22:29 UTC
LW: 2 AF: 1
AF
I still think the original proposal has a gerrymandering problem (slightly different from the one in my comment).

Say the AI is going to construct a sub-agent B (and the original AI will then go to sleep for the rest of the game, unable to view anything that happens). In the first turn, B will get a number of resources between 1 and 10. These resources are only useful for making paperclips. In the second turn, B can press a paperclips-to-staples button, which changes the original AI’s utility function to staple-maximization. In the third turn, B may create a number of paperclips equal to the number of resources, and may independently create 42 paperclips.

The naive strategy is to create a subagent that doesn’t press the button, then makes paperclips. This policy gets a reward of 5.5 (the average number of resources).

Consider an alternative strategy. The AI creates a sub-agent B with the following policy: if B gets between 1 and 5 resources, then it presses the paperclips-to-staples button; otherwise it doesn’t press the button. Then B will create the appropriate number of paperclips, or 42 staples, depending on the button.

Let u be number of paperclips produced, v be number of staples produced. Under the policy, we have $E [u | u \to u] = 8$ . And $E [v | u \to v] = 5$ . Since the button gets pressed half the time, the AI’s expected reward is

$P (u \to u) E [u | u \to u] + P (u \to v) (E [v | u \to v] + E [u | u \to u] - E [v | u \to v]) = 1 / 2 \cdot 8 + 1 / 2 (42 + 8 - 42) = 8$

Meanwhile, the AI’s expected reward if it never presses the button and just makes paperclips is 5.5. So it seems that the original proposal has a gerrymandering problem.

It’s possible that your new proposal fixes this problem, but first I want to check that this is actually a problem for the original proposal.
- Stuart_Armstrong 6 May 2016 15:32 UTC
  LW: 1 AF: 1
  AF Parent
  This is my punishment for being sloppy with definitions :-(
  
  Ok, let’s be clearer. When the button is pressed, a process begins, to change $u$ into $v$ . But the process is stocastic, and has a small chance of failing. The failure of this is designated by $u \to u$ .
  
  So if Pr refers to pressing the button, the expected reward is:
  
  $P (\neg P r) E (u | \neg P r) + P (P r, u \to u) E (u | P r, u \to u) + P (P r, u \to v) (E (v | P r, u \to v) + E (u | P r, u \to u) - E (v | P r, u \to v)) = P (\neg P r) E (u | \neg P r) + (P (P r, u \to u) + P (P r, u \to v)) E (u | P r, u \to u) = (1 / 2) 3 + (1 / 2) 8 = 5.5.$
  - jessicata 6 May 2016 19:33 UTC
    LW: 2 AF: 1
    AF Parent
    This makes more sense, thanks for explaining!
    
    I’m going to translate this to my notation so I understand it better (feel free to skip this, it’s mostly for my benefit). Let $X$ be the event that the coin flip succeeds (which usually happens). We can say that $X$ still has a value even if $\neg P r$ (its value just doesn’t do anything). So $u \to v$ happens iff $P r \land X$ .
    
    The expected utility is then
    
    $E [[\neg P r \lor \neg X] u + [P r \land X] (v + E [u | P r \land \neg X] - E [v | P r \land X])$
    
    Suppose the AI never updates its beliefs (because it goes into stasis the first round). Then this is
    
    $E [[\neg P r \lor \neg X] u] + E [[P r \land X] v] + P (P r \land X) E [u | P r \land \neg X] - E [[P r \land X] v]$ $= E [[\neg P r \lor \neg X] u] + P (P r \land X) E [u | P r \land \neg X]$ $= P (\neg P r) E [u | \neg P r, \neg X] + P (P r \land X) E [u | P r, \neg X]$ $\approx P (\neg P r) E [u | \neg P r, \neg X] + P (P r) E [u | P r, \neg X]$ $= E [u | \neg X]$ $= E [u | d o (\neg X)]$ (I wrote it as a causal counterfactual in the last step; it doesn’t make a difference since $X$ has no causal ancestors, but I find it easier to reason about this way)
    
    So in the end, if the whole game is to construct a subagent and then go into stasis, then the agent is just a $u$ -maximizer who believes $\neg X$ (as you said). It avoids the gerrymandering problem by doing a causal counterfactual.
    
    I think another way of framing this is that the button always succeeds, and agent optimizes $E [u | d o (\neg P r)]$ (which relates this back to the causal version of factoring out effects, where you optimize $q (P r) E [v | d o (P r)] + q (\neg P r) E [u | d o (\neg P r)]$ ; you get utility indifference with stasis by setting $q (P r) = 0$ ). This will be slightly different in that here the agent “believes” it will be magically prevented from pressing the button, whereas originally the agent just “believes” that the button will fail.
- Stuart_Armstrong 6 May 2016 16:16 UTC
  0 points
  AF Parent
  Note there is one way in which the AI functions as an outcome pump: it will accept bets, at any odds, against ever becoming a $v$ -maximiser. That’s because that’s what a pure $u$ -maximiser would do, and this agent design behave like a pure $u$ -maximiser.
- orthonormal 6 May 2016 1:25 UTC
  0 points
  AF Parent
  Typo: in the paragraph before the equation arrays, you forgot to change $E [v | u \to v]$ from 5 to 42 (you did so in the following equation arrays). This buffaloed me for a bit!
  - jessicata 6 May 2016 6:32 UTC
    0 points
    AF Parent
    Fixed, thanks.