There is not an easy patch because AIXI is defined as the optimal policy for a belief distribution over its hypothesis class, but we don’t really know how to talk about optimality for embedded agents (so the expectimax tree definition of AIXI cannot be easily extended to handle embeddedness).
I think there are two (possibly interrelated) things going on here. The first is that AIXI is formulated as an ideal process, including perfect Bayesianism, that is simply too computationally expensive to fit inside its environment: it’s in a higher complexity class. Any practical implementation will of course be an approximation computable by a Turing machine that can exist inside its environment. The second is that if approximation-AIXI’s world model includes an approximate model of itself, then (as your discussion of the anvil problem demonstrates), it’s not actually very hard for AIXI to reason about the likely effects of actions that decrease its computational capacity. But is cannot accurately model the effect of self-upgrades that significantly increase its computational capacity. Rough approximations like scaling laws can presumably be found, but it cannot answer questions like “If I upgraded myself to be 10 times smarter, and then new me did the same recursively another N times, how much better outcomes would my inproved approximations to ideal-AIXI produce?” There’s a Singularity-like effect here (in the SF author sense).
Yeah, I think you make a good point that increases in intelligence may be harder to understand than decreases, so settling whether this version of AIXI can pursue additional computational resources is an interesting open question.
I think there are two (possibly interrelated) things going on here. The first is that AIXI is formulated as an ideal process, including perfect Bayesianism, that is simply too computationally expensive to fit inside its environment: it’s in a higher complexity class. Any practical implementation will of course be an approximation computable by a Turing machine that can exist inside its environment. The second is that if approximation-AIXI’s world model includes an approximate model of itself, then (as your discussion of the anvil problem demonstrates), it’s not actually very hard for AIXI to reason about the likely effects of actions that decrease its computational capacity. But is cannot accurately model the effect of self-upgrades that significantly increase its computational capacity. Rough approximations like scaling laws can presumably be found, but it cannot answer questions like “If I upgraded myself to be 10 times smarter, and then new me did the same recursively another N times, how much better outcomes would my inproved approximations to ideal-AIXI produce?” There’s a Singularity-like effect here (in the SF author sense).
Yeah, I think you make a good point that increases in intelligence may be harder to understand than decreases, so settling whether this version of AIXI can pursue additional computational resources is an interesting open question.
I was saying that increases are harder than decreases.
typo, fixed now.
I suspected as much.