If a counterfactual version of A is told outright that O(A)==1, and yet sees a provable way to make O(A)==2, how do you justify not going crazy?
It’s not the correct way of interpreting observations, you shouldn’t let observations drive you crazy. Here, we have A’s action-definition that is given in factorized form: action=A(O(“A”)). Normally, you’d treat such decompositions as explicit dependence bias, and try substituting everything in before starting to reason about what would happen if. But if O(“A”) is an observation, then you’re not deciding action, that is A(O(“A”)). Instead, you’re deciding just A(-), an Observations → Actions map. So being told that you’ve observed “no award” doesn’t mean that you now know that O(“A”)=”no award”. It just means that you’re the subagent responsible for deciding a response to parameter “no award” in the strategy for A(-). You might also want to acausally coordinate with the subagent that is deciding the other part of that same strategy, a response to “award”.
And this all holds even if the agent knows what O(“A”) means, it would just be a bad idea to not include O(“A”) as part of the agent in that case, and so optimize the overall A(O(“A”)) instead of the smaller A(-).
At this point it seems we’re arguing over how to better formalize the original problem. The post asked what you should reply to Omega. Your reformulation asks what counterfactual-you should reply to counterfactual-Omega that doesn’t even have to say the same thing as the original Omega, and whose judgment of you came from the counterfactual void rather than from looking at you. I’m not sure this constitutes a fair translation. Some of the commenters here (e.g. prase) seem to intuitively lean toward my interpretation—I agree it’s not UDT-like, but think it might turn out useful.
At this point it seems we’re arguing over how to better formalize the original problem.
It’s more about making more explicit the question of what are observations, and what are boundaries of the agent (Which parts of the past lightcone are part of you? Just the cells in the brain? Why is that?), in deterministic decision problems. These were never explicitly considered before in the context of UDT. The problem statement states that something is “observation”, but we lack a technical counterpart of that notion. Your questions resulted from treating something that’s said to be an “observation” as epistemically relevant, writing knowledge about state of the territory which shouldn’t be logically transparent right into agent’s mind.
(Observations, possible worlds, etc. will very likely be the topic of my next post on ADT, once I resolve the mystery of observational knowledge to my satisfaction.)
Thanks, this looks like a fair summary (though a couple levels too abstract for my liking, as usual).
A note on epistemic relevance. Long ago, when we were just starting to discuss Newcomblike problems, the preamble usually went something like this: “Omega appears and somehow convinces you that it’s trustworthy”. So I’m supposed to listen to Omega’s words and somehow split them into an “epistemically relevant” part and an “observation” part, which should never mix? This sounds very shady. I hope we can disentangle this someday.
Your reformulation asks what counterfactual-you should reply to counterfactual-Omega that doesn’t even have to say the same thing as the original Omega.
Yes. If the agent doesn’t know what Omega actually says, this can be an important consideration (decisions are made by considering agent-provable properties of counterfactuals, all of which except the actual one are inconsistent, but not agent-inconsistent). If Omega’s decision is known (and not just observed), it just means that counterfactual-you’s response to counterfactual-Omega doesn’t control utility and could well be anything. But at this point I’m not sure in what sense anything can actually be logically known, and not in some sense just observed.
It’s not the correct way of interpreting observations, you shouldn’t let observations drive you crazy. Here, we have A’s action-definition that is given in factorized form: action=A(O(“A”)). Normally, you’d treat such decompositions as explicit dependence bias, and try substituting everything in before starting to reason about what would happen if. But if O(“A”) is an observation, then you’re not deciding action, that is A(O(“A”)). Instead, you’re deciding just A(-), an Observations → Actions map. So being told that you’ve observed “no award” doesn’t mean that you now know that O(“A”)=”no award”. It just means that you’re the subagent responsible for deciding a response to parameter “no award” in the strategy for A(-). You might also want to acausally coordinate with the subagent that is deciding the other part of that same strategy, a response to “award”.
And this all holds even if the agent knows what O(“A”) means, it would just be a bad idea to not include O(“A”) as part of the agent in that case, and so optimize the overall A(O(“A”)) instead of the smaller A(-).
At this point it seems we’re arguing over how to better formalize the original problem. The post asked what you should reply to Omega. Your reformulation asks what counterfactual-you should reply to counterfactual-Omega that doesn’t even have to say the same thing as the original Omega, and whose judgment of you came from the counterfactual void rather than from looking at you. I’m not sure this constitutes a fair translation. Some of the commenters here (e.g. prase) seem to intuitively lean toward my interpretation—I agree it’s not UDT-like, but think it might turn out useful.
It’s more about making more explicit the question of what are observations, and what are boundaries of the agent (Which parts of the past lightcone are part of you? Just the cells in the brain? Why is that?), in deterministic decision problems. These were never explicitly considered before in the context of UDT. The problem statement states that something is “observation”, but we lack a technical counterpart of that notion. Your questions resulted from treating something that’s said to be an “observation” as epistemically relevant, writing knowledge about state of the territory which shouldn’t be logically transparent right into agent’s mind.
(Observations, possible worlds, etc. will very likely be the topic of my next post on ADT, once I resolve the mystery of observational knowledge to my satisfaction.)
Thanks, this looks like a fair summary (though a couple levels too abstract for my liking, as usual).
A note on epistemic relevance. Long ago, when we were just starting to discuss Newcomblike problems, the preamble usually went something like this: “Omega appears and somehow convinces you that it’s trustworthy”. So I’m supposed to listen to Omega’s words and somehow split them into an “epistemically relevant” part and an “observation” part, which should never mix? This sounds very shady. I hope we can disentangle this someday.
Yes. If the agent doesn’t know what Omega actually says, this can be an important consideration (decisions are made by considering agent-provable properties of counterfactuals, all of which except the actual one are inconsistent, but not agent-inconsistent). If Omega’s decision is known (and not just observed), it just means that counterfactual-you’s response to counterfactual-Omega doesn’t control utility and could well be anything. But at this point I’m not sure in what sense anything can actually be logically known, and not in some sense just observed.