First, to clear up a possible confusion, the S in my P is not supposed to be a variable. It’s a constant, more specifically a piece of code that implements UDT1 itself. (If I sometimes talk about it as if it’s a variable, that’s because I’m trying to informally describe what is going on inside the computation that UDT1 does.)
For the more general question of how do we know the structure of the world program, the idea is that for an actual AI, we would program it to care about all possible world programs (or more generally, mathematical structures, see example 3 in my UDT1 post, but also Nesov’s recent post for a critique). The implementation of UDT1 in the AI would then figure out which world programs it’s in by looking at its inputs (which would contain all of the AI’s memories and sensory data) and checking which world programs call it with those inputs.
For these sample problems, the assumption is that somehow Omega has previously provided us with enough evidence for us to trust its word on what the structure of the current problem is. So in the actual P, ‘S(i, “box contains $1M”)’ is really something like ‘S(memories, omegas_explanations_about_this_problem, i, “box contains $1M”)’ and these additional inputs allow S to conclude that it’s being invoked inside this P, and not some other world program.
(An additional subtlety here is that if we consider all possible world programs, there are bound to be some other world programs where S is being called with these exact same inputs, for example ones where S is being instantiated inside a Boltzmann brain, but presumably those worlds/regions have very low weights, meaning that the AI doesn’t care much about them.)
Let me know if that answers your questions/concerns. I didn’t answer you point by point because I’m not sure which questions/concerns remain after you see my general answers. Feel free to repeat anything you still want me to answer.
First, to clear up a possible confusion, the S in my P is not supposed to be a variable. It’s a constant, more specifically a piece of code that implements UDT1 itself. (If I sometimes talk about it as if it’s a variable, that’s because I’m trying to informally describe what is going on inside the computation that UDT1 does.)
Then it should be S(P), because S can’t make any decisions without getting to read the problem description.
Note that since our agent is considering possible world-programs, these world-programs are in some sense already part of the agent’s program (and the agent is in turn part of some of these world-programs-inside-the-agent, which reflects recursive character of the definition of the agent-program). The agent is a much better top-level program to consider than all-possible-world-programs, which is even more of a simplification if these world-programs somehow “exist at the same time”. When the (prior) definition of the world is seen as already part of the agent, a lot of the ontological confusion goes away.
First, to clear up a possible confusion, the S in my P is not supposed to be a variable. It’s a constant, more specifically a piece of code that implements UDT1 itself. (If I sometimes talk about it as if it’s a variable, that’s because I’m trying to informally describe what is going on inside the computation that UDT1 does.)
For the more general question of how do we know the structure of the world program, the idea is that for an actual AI, we would program it to care about all possible world programs (or more generally, mathematical structures, see example 3 in my UDT1 post, but also Nesov’s recent post for a critique). The implementation of UDT1 in the AI would then figure out which world programs it’s in by looking at its inputs (which would contain all of the AI’s memories and sensory data) and checking which world programs call it with those inputs.
For these sample problems, the assumption is that somehow Omega has previously provided us with enough evidence for us to trust its word on what the structure of the current problem is. So in the actual P, ‘S(i, “box contains $1M”)’ is really something like ‘S(memories, omegas_explanations_about_this_problem, i, “box contains $1M”)’ and these additional inputs allow S to conclude that it’s being invoked inside this P, and not some other world program.
(An additional subtlety here is that if we consider all possible world programs, there are bound to be some other world programs where S is being called with these exact same inputs, for example ones where S is being instantiated inside a Boltzmann brain, but presumably those worlds/regions have very low weights, meaning that the AI doesn’t care much about them.)
Let me know if that answers your questions/concerns. I didn’t answer you point by point because I’m not sure which questions/concerns remain after you see my general answers. Feel free to repeat anything you still want me to answer.
Then it should be S(P), because S can’t make any decisions without getting to read the problem description.
Note that since our agent is considering possible world-programs, these world-programs are in some sense already part of the agent’s program (and the agent is in turn part of some of these world-programs-inside-the-agent, which reflects recursive character of the definition of the agent-program). The agent is a much better top-level program to consider than all-possible-world-programs, which is even more of a simplification if these world-programs somehow “exist at the same time”. When the (prior) definition of the world is seen as already part of the agent, a lot of the ontological confusion goes away.