I don’t really understand how this could occur in a TDT-agent. The agent’s algorithm is causally dependent on ‘(max $5 $10), but considering the counterfactual severs that dependence. Observing a money-optimizer (let’s call it B) choosing $5 over $10 would presumably cause the agent (call it A) to update its model of B to no longer depend on ’(max $5 $10). Am I missing something here?
Correctly getting to the comparison of $5 and $10 is the whole point of the exercise. An agent is trying to evaluate the consequences of its action, A, which is defined by agent’s algorithm and is not known explicitly in advance. To do that, it could in some sense consider hypotheticals where its action assumes its possible values. One such hypothetical could involve a claim that A=$5. The error in question is about looking at the claim that A=$5 and making incorrect conclusions (which would result in an action that doesn’t depend on comparing $5 and $10).
I don’t really understand how this could occur in a TDT-agent. The agent’s algorithm is causally dependent on ‘(max $5 $10), but considering the counterfactual severs that dependence. Observing a money-optimizer (let’s call it B) choosing $5 over $10 would presumably cause the agent (call it A) to update its model of B to no longer depend on ’(max $5 $10). Am I missing something here?
Correctly getting to the comparison of $5 and $10 is the whole point of the exercise. An agent is trying to evaluate the consequences of its action, A, which is defined by agent’s algorithm and is not known explicitly in advance. To do that, it could in some sense consider hypotheticals where its action assumes its possible values. One such hypothetical could involve a claim that A=$5. The error in question is about looking at the claim that A=$5 and making incorrect conclusions (which would result in an action that doesn’t depend on comparing $5 and $10).