Edit: Come to think of it, your response would probably start with something like “I know that, but...”, but I may have addressed some of what your further objections would have been in my replay to Cyan.
...
Counting that as “identifying itself with the approximation” seems overly generous, but if we grant that, I still don’t see any reason that AIXI would end up considering such a hypothesis likely, or that it would be likely to understand any mechanism for it to self-modify correctly in terms of its model of modifying an external approximation to itself.
The simple answer is: because these models accurately predict the observations after self-modification actions are performed. Of course, the problem is not that simple, because of the non-ergodicity issues I’ve discussed before: when you fiddle with your source code without knowing what you are doing is easy to accidentally ‘drop an anvil’ on yourself. But this is a hard problem without any simple solution IMHO.
The simple answer is: because these models accurately predict the observations after self-modification actions are performed.
For that to be true, the environment has to keep sending AIXI the same signals that it sends the approximation even after it stops paying attention to AIXI’s output. Even in that case, the fact that this model correctly predicts future observations doesn’t help at all. Prior to self-modifying, AIXI does not have access to information about what it will observe after self-modifying.
I agree with you that the non-ergodicity issues don’t have any simple solution. I haven’t been making a big deal about non-ergodicity because there don’t exist any agents that perform optimally in all non-ergodic environments (since one computable environment can permanently screw you for doing one thing, and another computable environment can permanently screw you for doing anything else), so it’s not a problem specific to AIXI-like agents, and AIXI actually seems like it should act fairly reasonably in non-ergodic computable environments separated from the agent by a Cartesian barrier, given the information available to it.
Then I don’t think we actually disagree. I mean, it was well known that the AIXI proof of optimality requred ergodicity, since the original Hutter’s paper.
...
The simple answer is: because these models accurately predict the observations after self-modification actions are performed.
Of course, the problem is not that simple, because of the non-ergodicity issues I’ve discussed before: when you fiddle with your source code without knowing what you are doing is easy to accidentally ‘drop an anvil’ on yourself. But this is a hard problem without any simple solution IMHO.
For that to be true, the environment has to keep sending AIXI the same signals that it sends the approximation even after it stops paying attention to AIXI’s output. Even in that case, the fact that this model correctly predicts future observations doesn’t help at all. Prior to self-modifying, AIXI does not have access to information about what it will observe after self-modifying.
I agree with you that the non-ergodicity issues don’t have any simple solution. I haven’t been making a big deal about non-ergodicity because there don’t exist any agents that perform optimally in all non-ergodic environments (since one computable environment can permanently screw you for doing one thing, and another computable environment can permanently screw you for doing anything else), so it’s not a problem specific to AIXI-like agents, and AIXI actually seems like it should act fairly reasonably in non-ergodic computable environments separated from the agent by a Cartesian barrier, given the information available to it.
Then I don’t think we actually disagree.
I mean, it was well known that the AIXI proof of optimality requred ergodicity, since the original Hutter’s paper.