I usually understand “mind-reading” to encompass being aware of the current state of a system. Two systems that simply know one another’s strategies can’t predict one another’s behaviors if their strategies include random coin flips, for example, or depend on information that one system can observe but the other cannot; whereas I would expect telepathic agents to be aware of the results of such observations as well.
If you did have two telepaths playing any game, and one of them decided a mixed strategy was optimal, they wouldn’t want to know what action they were playing until it was played- because otherwise they might leak that knowledge to the other player. That is, in a competitive situation I don’t think mind-reading would extend to coin-reading, but if your understanding is common then ‘mind-reading’ is a bad phrase to use. Is there a good word for “has access to its opponent’s source code”? Bonus points if it starts with a T.
(Also, my understanding is that TDT will defect against any mixed strategy in the prisoner’s dilemma.)
(Also, my understanding is that TDT will defect against any mixed strategy in the prisoner’s dilemma.)
Not necessarily. It will play a mixed strategy against an identical mixed strategy if that if what it needs to do to get them to play mixed rather than D. It’s the other guy being weird and arbitrary in that case, not the TDT.
Well, it certainly will defect against any mixed strategy that is hard coded into the opponent’s source code. On the other hand, if the mixed strategy the opponent plays is dependent on what it predicts the TDT agent will play, then the TDT agent will figure out which outcome has a higher expected utility:
(I defect, Opponent runs “defection predicted” mixed strategy) (I cooperate, Opponent runs “cooperation detected” mixed strategy)
Of course, this is still simplifying things a bit, since it assumes that the opponent can perfectly predict one’s strategy, and it also rules out the possibility of the TDT agent using a mixed strategy himself.
Thus the actual computation is more like ArgMax(Sum(ExpectedUtility(S,T)*P(T|S)))
where the argmax is over S: all possible mixed strategies for the TDT agent the sum is over T: all possible mixed strategies for the opponent and P(T|S) is the probability that opponent will play T, given that we choose to play S. (so this is essentially an estimate of the opponent’s predictive power.)
Won’t that let the opponent steal utility from you? Consider the case where you’re going up against another TDTer which is willing to consider both the strategy “if they cooperate only if I cooperate, then cooperate with 99% probability” and “if they cooperate only if I cooperate, then cooperate.” You want your response to the first strategy to be defection and your response to the second strategy to be cooperation, so it’s in their interests to play the second strategy.
You’re right, if the opponent is a TDT agent. I was assuming that the opponent was simply a prediction=>mixed strategy mapper. (In fact, I always thought that the strategy 51% one-box 49% two box would game the system, assuming that Omega just predicts the outcome which is most likely).
If the opponent is a TDT agent, then it becomes more complex, as in the OP. Just as above, you have to take the argmax over all possible y->x mappings, instead of simply taking the argmax over all outputs.
Putting it in that perspective, essentially in this case we are adding all possible mixed strategies to the space of possible outputs. Hmmm… That’s somewhat a better way of putting it than everything else I said.
In any case, two TDT agents will both note that the program which only cooperates 100% iff the opponent cooperates 100% dominates all other mixed strategies against such an opponent.
So to answer the original question: Yes, it will defect against blind mixed strategies. No, it will not necessarily defect against simple (prediction =>mixed strategy) mappers. N/A against another TDT agent, as neither will ever play a mixed strategy, so to ask what whether it would cooperate with a mixed strategy TDT agent is counterfactual.
EDIT: Thinking some more, I realize that TDT agents will consider the sort of 99% rigging against each other — and will find that it is better than the cooperate IFF strategy. However, this is where the “sanity check” become important. The TDT agent will realize that although such a pure agent would do better against a TDT opponent, the opponent knows that you are a TDT agent as well, and thus will not fall for the trap.
Out of this I’ve reached two conclusions:
The sanity check outlined above is not broad enough, as it only sanity checks the best agents, whereas even if the best possible agent fails the sanity check, there still could be an improvement over the nash equilibrium which passes.
Eliezer’s previous claim that a TDT agent will never regret being a TDT agent given full information is wrong (hey, I thought it was right too). Either it gives in to a pure 99% rigger or it does not. If it does, then it regrets not being able to 99% rig another TDT agent. If it does not, then it regrets not being a simple hard-coded cooperator against a 99% rigger. This probably could be formalized a bit more, but I’m wondering if Eliezer et. al. have considered this?
EDIT2: I realize I was a bit confused before. Feeling a bit stupid. Eliezer never claimed that a TDT agent won’t regret being a TDT agent (which is obviously possible, just consider a clique-bot opponent), but that a TDT agent will never regret being given information.
(In fact, I always thought that the strategy 51% one-box 49% two box would game the system, assuming that Omega just predicts the outcome which is most likely).
Incidentally, my preferred version of Newcomb is that if the Predictor decides that your chance of one-boxing is p, it puts (one million times p) dollars in the big box. Presumably, you know that the Predictor is both extremely well-calibrated and shockingly accurate (it usually winds up with p near 0 or near 1).
The sanity check outlined above is not broad enough, as it only sanity checks the best agents, whereas even if the best possible agent fails the sanity check, there still could be an improvement over the nash equilibrium which passes.
Yup, this is where I’m going in a future post. See the footnote on this post about other variants of TDT; there’s a balance between missing workable deals against genuinely stubborn opponents, and failing to get the best possible deal from clever but flexible opponents. (And, if I haven’t made a mistake in the reasoning I haven’t checked, there is a way to use further cleverness to do still better.)
For now, note that TDT wouldn’t necessarily prefer to be a hard-coded 99% cooperator in general, since those get “screw you” mutual defections from some (stubborn) agents that mutually cooperate with TDT.
You’ve made it into a bargaining game with that mixed strategy, and indeed the version of TDT we introduced will defect against an opponent that outputs the mixed strategy (if that opponent would output a pure cooperate if that were the only way to get its adversary to cooperate). But bargaining games are complicated, and I’m saving that for another post.
I usually understand “mind-reading” to encompass being aware of the current state of a system. Two systems that simply know one another’s strategies can’t predict one another’s behaviors if their strategies include random coin flips, for example, or depend on information that one system can observe but the other cannot; whereas I would expect telepathic agents to be aware of the results of such observations as well.
If you did have two telepaths playing any game, and one of them decided a mixed strategy was optimal, they wouldn’t want to know what action they were playing until it was played- because otherwise they might leak that knowledge to the other player. That is, in a competitive situation I don’t think mind-reading would extend to coin-reading, but if your understanding is common then ‘mind-reading’ is a bad phrase to use. Is there a good word for “has access to its opponent’s source code”? Bonus points if it starts with a T.
(Also, my understanding is that TDT will defect against any mixed strategy in the prisoner’s dilemma.)
Not necessarily. It will play a mixed strategy against an identical mixed strategy if that if what it needs to do to get them to play mixed rather than D. It’s the other guy being weird and arbitrary in that case, not the TDT.
Well, it certainly will defect against any mixed strategy that is hard coded into the opponent’s source code. On the other hand, if the mixed strategy the opponent plays is dependent on what it predicts the TDT agent will play, then the TDT agent will figure out which outcome has a higher expected utility:
(I defect, Opponent runs “defection predicted” mixed strategy)
(I cooperate, Opponent runs “cooperation detected” mixed strategy)
Of course, this is still simplifying things a bit, since it assumes that the opponent can perfectly predict one’s strategy, and it also rules out the possibility of the TDT agent using a mixed strategy himself.
Thus the actual computation is more like
ArgMax(Sum(ExpectedUtility(S,T)*P(T|S)))
where the argmax is over S: all possible mixed strategies for the TDT agent
the sum is over T: all possible mixed strategies for the opponent
and P(T|S) is the probability that opponent will play T, given that we choose to play S. (so this is essentially an estimate of the opponent’s predictive power.)
Won’t that let the opponent steal utility from you? Consider the case where you’re going up against another TDTer which is willing to consider both the strategy “if they cooperate only if I cooperate, then cooperate with 99% probability” and “if they cooperate only if I cooperate, then cooperate.” You want your response to the first strategy to be defection and your response to the second strategy to be cooperation, so it’s in their interests to play the second strategy.
You’re right, if the opponent is a TDT agent. I was assuming that the opponent was simply a prediction=>mixed strategy mapper. (In fact, I always thought that the strategy 51% one-box 49% two box would game the system, assuming that Omega just predicts the outcome which is most likely).
If the opponent is a TDT agent, then it becomes more complex, as in the OP. Just as above, you have to take the argmax over all possible y->x mappings, instead of simply taking the argmax over all outputs.
Putting it in that perspective, essentially in this case we are adding all possible mixed strategies to the space of possible outputs. Hmmm… That’s somewhat a better way of putting it than everything else I said.
In any case, two TDT agents will both note that the program which only cooperates 100% iff the opponent cooperates 100% dominates all other mixed strategies against such an opponent.
So to answer the original question: Yes, it will defect against blind mixed strategies. No, it will not necessarily defect against simple (prediction =>mixed strategy) mappers. N/A against another TDT agent, as neither will ever play a mixed strategy, so to ask what whether it would cooperate with a mixed strategy TDT agent is counterfactual.
EDIT: Thinking some more, I realize that TDT agents will consider the sort of 99% rigging against each other — and will find that it is better than the cooperate IFF strategy. However, this is where the “sanity check” become important. The TDT agent will realize that although such a pure agent would do better against a TDT opponent, the opponent knows that you are a TDT agent as well, and thus will not fall for the trap.
Out of this I’ve reached two conclusions:
The sanity check outlined above is not broad enough, as it only sanity checks the best agents, whereas even if the best possible agent fails the sanity check, there still could be an improvement over the nash equilibrium which passes.
Eliezer’s previous claim that a TDT agent will never regret being a TDT agent given full information is wrong (hey, I thought it was right too). Either it gives in to a pure 99% rigger or it does not. If it does, then it regrets not being able to 99% rig another TDT agent. If it does not, then it regrets not being a simple hard-coded cooperator against a 99% rigger. This probably could be formalized a bit more, but I’m wondering if Eliezer et. al. have considered this?
EDIT2: I realize I was a bit confused before. Feeling a bit stupid. Eliezer never claimed that a TDT agent won’t regret being a TDT agent (which is obviously possible, just consider a clique-bot opponent), but that a TDT agent will never regret being given information.
Incidentally, my preferred version of Newcomb is that if the Predictor decides that your chance of one-boxing is p, it puts (one million times p) dollars in the big box. Presumably, you know that the Predictor is both extremely well-calibrated and shockingly accurate (it usually winds up with p near 0 or near 1).
Yup, this is where I’m going in a future post. See the footnote on this post about other variants of TDT; there’s a balance between missing workable deals against genuinely stubborn opponents, and failing to get the best possible deal from clever but flexible opponents. (And, if I haven’t made a mistake in the reasoning I haven’t checked, there is a way to use further cleverness to do still better.)
For now, note that TDT wouldn’t necessarily prefer to be a hard-coded 99% cooperator in general, since those get “screw you” mutual defections from some (stubborn) agents that mutually cooperate with TDT.
You’ve made it into a bargaining game with that mixed strategy, and indeed the version of TDT we introduced will defect against an opponent that outputs the mixed strategy (if that opponent would output a pure cooperate if that were the only way to get its adversary to cooperate). But bargaining games are complicated, and I’m saving that for another post.