Ok, I think I see where our formalizations differ. In the formalization I’m using, the decision theory produces a strategy, which is a function that’s given as an argument to the world program. The world program invokes the strategy zero or more times, each time passing in some arguments that give whatever information is available to the agent at some point, and getting back a (real or predicted) decision. The world program is completely self-contained; other than through the argument it receives, it may not contain references to the agent’s choices at all. The strategy is similarly self-contained; it receives no information about the world except through the arguments the world program passes to it. Then separately from that, a “decision theory” is a function that takes a symbolic representation of a world program, and returns a symbolic representation of a strategy.
Ultimately, this amounts to a refactoring; results that hold in one system still hold in the other, if you map the definitions appropriately. However, I’ve found that structuring problems this way makes the theory easier to build on, and makes underspecified problems easier to notice.
The world program is completely self-contained; other than through the argument it receives, it may not contain references to the agent’s choices at all.
Can you formalize this requirement? If I copy agent’s code, rename all symbols, obfuscate it, simulate its execution in a source code interpreter that runs in a hardware emulator running on an emulated linux box running on javascript inside a browser running on Windows running on a hardware simulator implemented (and then obfuscated again) in the same language as the world program, and insert this thing in the world program (along with a few billion people and a planet and a universe), how can you possibly make sure that there is no dependence?
Can you formalize this requirement? If I copy agent’s code … and insert this thing in the world program, how can you possibly make sure that there is no dependence?
You don’t get to do that, because when you’re writing World, the Strategy hasn’t been determined yet. Think of it as a challenge-response protocol; World is a challenge, and Strategy is a response. You can still do agent-copying, but you have to enlarge the scope of World to include the rules by which that copying was done, or else you get unrelated agents instead of copies.
To copy agent’s code, you don’t need to know strategy. World naturally changes if you change it, and the strategy might change as well if you run the agent on a changed world, but agent’s code is still the same, and you know this code. The new world will only depend on the new strategy, not the old one, but now we have a world that depends on its agent’s strategy, and you won’t be able to find how it does, if you don’t already know.
In any case, all this copying is irrelevant, because the point is that there can exist very convoluted worlds that depend on agent’s action, but it’s not feasible to know that they do or how they do. And we don’t get to choose the real world.
Ok, I think I see where our formalizations differ. In the formalization I’m using, the decision theory produces a strategy, which is a function that’s given as an argument to the world program. The world program invokes the strategy zero or more times, each time passing in some arguments that give whatever information is available to the agent at some point, and getting back a (real or predicted) decision. The world program is completely self-contained; other than through the argument it receives, it may not contain references to the agent’s choices at all. The strategy is similarly self-contained; it receives no information about the world except through the arguments the world program passes to it. Then separately from that, a “decision theory” is a function that takes a symbolic representation of a world program, and returns a symbolic representation of a strategy.
Ultimately, this amounts to a refactoring; results that hold in one system still hold in the other, if you map the definitions appropriately. However, I’ve found that structuring problems this way makes the theory easier to build on, and makes underspecified problems easier to notice.
Can you formalize this requirement? If I copy agent’s code, rename all symbols, obfuscate it, simulate its execution in a source code interpreter that runs in a hardware emulator running on an emulated linux box running on javascript inside a browser running on Windows running on a hardware simulator implemented (and then obfuscated again) in the same language as the world program, and insert this thing in the world program (along with a few billion people and a planet and a universe), how can you possibly make sure that there is no dependence?
You don’t get to do that, because when you’re writing World, the Strategy hasn’t been determined yet. Think of it as a challenge-response protocol; World is a challenge, and Strategy is a response. You can still do agent-copying, but you have to enlarge the scope of World to include the rules by which that copying was done, or else you get unrelated agents instead of copies.
To copy agent’s code, you don’t need to know strategy. World naturally changes if you change it, and the strategy might change as well if you run the agent on a changed world, but agent’s code is still the same, and you know this code. The new world will only depend on the new strategy, not the old one, but now we have a world that depends on its agent’s strategy, and you won’t be able to find how it does, if you don’t already know.
In any case, all this copying is irrelevant, because the point is that there can exist very convoluted worlds that depend on agent’s action, but it’s not feasible to know that they do or how they do. And we don’t get to choose the real world.