Actually, I’m not sure this matters. If the simulated agent knows he’s not getting a reward, he’d still want to choose so that the nonsimulated version of himself gets the best reward.
So the problem is that the best answer is unavailable to the simulated agent: in the simulation you should one box and in the ‘real’ problem you’d like to two box, but you have no way of knowing whether you’re in the simulation or the real problem.
Agents that Omega didn’t simulate don’t have the problem of worrying whether they’re making the decision in a simulation or not, so two boxing is the correct answer for them.
The decisions being made are very different between an agent that has to make the decision twice and the first decision will affect the payoff of the second versus an agent that has to make the decision only once, so I think that in reality perhaps the problem does collapse down to an ‘unfair’ one because the TDT agent is presented with an essentially different problem to a nonTDT agent.
Actually, I’m not sure this matters. If the simulated agent knows he’s not getting a reward, he’d still want to choose so that the nonsimulated version of himself gets the best reward.
So the problem is that the best answer is unavailable to the simulated agent: in the simulation you should one box and in the ‘real’ problem you’d like to two box, but you have no way of knowing whether you’re in the simulation or the real problem.
Agents that Omega didn’t simulate don’t have the problem of worrying whether they’re making the decision in a simulation or not, so two boxing is the correct answer for them.
The decisions being made are very different between an agent that has to make the decision twice and the first decision will affect the payoff of the second versus an agent that has to make the decision only once, so I think that in reality perhaps the problem does collapse down to an ‘unfair’ one because the TDT agent is presented with an essentially different problem to a nonTDT agent.