Tetraspace comments on Tetraspace Grouping’s Shortform

Tetraspace 28 May 2020 16:58 UTC
1 point
So the definition of myopia given in Defining Myopia was quite similar to my expansion in the But Wait There’s More section; you can roughly match them up by saying $r (x) = \sum_{i} f_{i} r_{i} (x)$ and $y_{i} (x) = (1 - f_{i}) r_{i} (x)$ , where $f_{i}$ is a real number corresponding to the amount that the agent cares about rewards obtained in episode $i$ and $r_{i}$ is the reward obtained in episode $i$ . Putting both of these into the sum gives $R (x) = \sum_{i} r_{i} (x)$ , the undiscounted, non-myopic reward that the agent eventually obtains.
In terms of the $R = R_{0} + R_{1}$ definition that I give in the uncertainty framing, this is $R_{0} = R (x, y_{0}) = \sum_{i} f_{i} r_{i} (x) + \sum_{i} (1 - f_{i}) r_{i} (x_{0})$ , and $R_{1} = R (x, y) - R (x, y_{0}) = \sum_{i} (1 - f_{i}) (r_{i} (x) - r_{i} (x_{0}))$ .
So if you let $r$ be a vector of the reward obtained on each step and $f$ be a vector of how much the agent cares about each step then $x \to x + ϵ \sum_{i} f_{i} \frac{\partial r_{i}}{\partial x}$ , and thus the change to the overall reward is $R \to R + ϵ \sum_{i} \frac{\partial r_{i}}{\partial x} \sum_{j} f_{j} \frac{\partial r_{j}}{\partial x}$ , which can be negative if the two sums have different signs.
I was hoping that a point would reveal itself to me about now but I’ll have to get back to you on that one.