Vanessa Kosoy comments on Vanessa Kosoy’s Shortform

[Vanessa Kosoy]

Maybe I am confused by what you mean by $S$. I thought it was the state space, but that isn’t consistent with $r$ in your post which was defined over $A\times O\to Q$?

I’m not entirely sure what you mean by the state space.$S$ is a state space associated specifically with the utility function. It has nothing to do with the state space of the environment. The reward function in the OP is $(A\times O{)}^{\ast }\to R$, not $A\times O\to R$. I slightly abused notation by defining $r:S\to Q$ in the parent comment. Let’s say it’s $r^{\prime }:S\to Q$ and $r$ is defined by using $T$ to translate the history to the (last) state and then applying $r^{\prime }$.

One more question, this one about the priors: what are they a prior over exactly? …I ask because the term $D_{KL}\left({\zeta }_0||\zeta \right)$ will be positive infinity if $\zeta$ is zero for any value where ${\zeta }_0$ is non-zero.

The prior is just an environment i.e. a partial mapping $\zeta :(A\times O{)}^{\ast }\to \Delta O$ defined on every history to which it doesn’t itself assign probability $0$. The expression $D_{KL}\left(\xi ||\zeta \right)$ means that we consider all possible ways to choose a Polish space $X$, probability distributions $\mu ,\nu \in \Delta X$ and a mapping $f:X\times (A\times O{)}^{\ast }\to \Delta O$ s.t.$\zeta =E_{\mu }\left[f\right]$ and $\xi =E_{\nu }\left[f\right]$ (where the expected value is defined using the Bayes law and not pointwise, see also the definition of “instrumental states” here), and take the minimum over all of them of $D_{KL}\left(\nu ||\mu \right)$.