Plain vanilla AIXI/AIXItl doesn’t have a reward register.
Yeah, I changed that while your reply was in progress.
More to come later...
ETA: Later is now!
I’m still having trouble understanding why you think that AIXI is optimal under violations of ergodicity.
I don’t think that AIXI is optimal under violations of ergodicity; I’m not talking about the optimality of AIXI at all. I’m talking about whether or not the Solomonoff induction part is capable of prompting AIXI to preseve itself.
I’m going to try to taboo “AIXI believes” and “AIXI thinks”. In hypothetical reality, the physically instantiated AIXI agent is a motherboard with sensors and actuators that are connected to the input and output pins, respectively, of a box labelled “Solomonoff Magic”. This agent is in a room. Somewhere in the space of all possible programs there are two programs. The first is just the maximally compressed version of the second, i.e., the first and the second give the same outputs on all possible inputs. The second one in written in Java, with a front-end interpreter that translates the Java program into the native language of the Solomonoff unit. (Java plus a prefix-free coding, blar blar blar). This program contains a human-readable physics simulation and an observation prediction routine. The initial conditions of the physics simulation match hypothetical reality except that the innards of the CPU are replaced by a computable approximation, including things like waste heat and whatnot. The simulation uses the input to determine the part of the initial conditions that specifies simulated-AIXI’s output voltages… ah! ah! ah! Found the Cartesian boundary! No matter how faithful the physics simulation is, AIXI only ever asks for one time-step at a time, so although the simulation’ state propagates to simulated-AIXI’s input voltages, it doesn’t propagate all the way through to the output voltage.
Thank you for your patience, Nate. The outside view wins again.
The simulation uses the input to determine the part of the initial conditions that specifies simulated-AIXI’s output voltages… ah! ah! ah! Found the Cartesian boundary! No matter how faithful the physics simulation is, AIXI only ever asks for one time-step at a time, so although the simulation’ state propagates to simulated-AIXI’s input voltages, it doesn’t propagate all the way through to the output voltage.
Actually, I find myself in a state of uncertainty as a result of doing a close reading section 2.6 of the Gentle Introduction to AIXI in light of your comment here. You quoted Paul Christiano as saying
Recall the definition of AIXI: A will try to infer a simple program which takes A’s outputs as input and provides A’s inputs as output, and then choose utility maximizing actions with respect to that program.
EY, Nate, Rob, and various commenters here (including myself until recently) all seemed to take this as given. For instance, above I wrote:
The simulation uses the input [i.e., action choice fed in as required by expectimax] to determine the part of the initial conditions that specifies simulated-AIXI’s output voltages [emphasis added]
On this “program-that-takes-action-choice-as-an-input” view (perhaps inspired by a picture like that on page 7 of the Gentle Introduction and surrounding text), a simulated event like, say, a laser cutter slicing AIXI’s (sim-)physical instantiation in half, could sever the (sim-)causal connection from (sim-)AIXI’s input wire to its output wire, and this event would not change the fact that the simulation specifies the voltage on the output wire from the expectimax action choice.
Your claim, if I understand you correctly, is that the AIXI formalism does not actually express this kind of back-and-forth state swapping. Rather, for any given universe-modeling program, it simulates forward from the specification of the (sim-)input wire voltage (or does something computationally equivalent), not from a specification of the (sim-)output wire voltage. There is some universe-model which simulates a computable approximation of all of (sim-)AIXI’s physical state changes; once the end state of has been specified, real-AIXI gives zero weight all branches of the expectimax tree that do not have an action that matches the state of (sim-)AIXI’s output wire.
Yeah, I changed that while your reply was in progress.
More to come later...
ETA: Later is now!
I don’t think that AIXI is optimal under violations of ergodicity; I’m not talking about the optimality of AIXI at all. I’m talking about whether or not the Solomonoff induction part is capable of prompting AIXI to preseve itself.
I’m going to try to taboo “AIXI believes” and “AIXI thinks”. In hypothetical reality, the physically instantiated AIXI agent is a motherboard with sensors and actuators that are connected to the input and output pins, respectively, of a box labelled “Solomonoff Magic”. This agent is in a room. Somewhere in the space of all possible programs there are two programs. The first is just the maximally compressed version of the second, i.e., the first and the second give the same outputs on all possible inputs. The second one in written in Java, with a front-end interpreter that translates the Java program into the native language of the Solomonoff unit. (Java plus a prefix-free coding, blar blar blar). This program contains a human-readable physics simulation and an observation prediction routine. The initial conditions of the physics simulation match hypothetical reality except that the innards of the CPU are replaced by a computable approximation, including things like waste heat and whatnot. The simulation uses the input to determine the part of the initial conditions that specifies simulated-AIXI’s output voltages… ah! ah! ah! Found the Cartesian boundary! No matter how faithful the physics simulation is, AIXI only ever asks for one time-step at a time, so although the simulation’ state propagates to simulated-AIXI’s input voltages, it doesn’t propagate all the way through to the output voltage.
Thank you for your patience, Nate. The outside view wins again.
Can you please expand?
Actually, I find myself in a state of uncertainty as a result of doing a close reading section 2.6 of the Gentle Introduction to AIXI in light of your comment here. You quoted Paul Christiano as saying
EY, Nate, Rob, and various commenters here (including myself until recently) all seemed to take this as given. For instance, above I wrote:
On this “program-that-takes-action-choice-as-an-input” view (perhaps inspired by a picture like that on page 7 of the Gentle Introduction and surrounding text), a simulated event like, say, a laser cutter slicing AIXI’s (sim-)physical instantiation in half, could sever the (sim-)causal connection from (sim-)AIXI’s input wire to its output wire, and this event would not change the fact that the simulation specifies the voltage on the output wire from the expectimax action choice.
Your claim, if I understand you correctly, is that the AIXI formalism does not actually express this kind of back-and-forth state swapping. Rather, for any given universe-modeling program, it simulates forward from the specification of the (sim-)input wire voltage (or does something computationally equivalent), not from a specification of the (sim-)output wire voltage. There is some universe-model which simulates a computable approximation of all of (sim-)AIXI’s physical state changes; once the end state of has been specified, real-AIXI gives zero weight all branches of the expectimax tree that do not have an action that matches the state of (sim-)AIXI’s output wire.
Do I have that about right?