I recall a paper written by a student of Scott Aaronson about an IPD tournament (mentioned in the article about Eigenmorality). Indeed the winners were agents that kept a model of the opponent and responded in kind: T-f-T wasn’t by far the optimal algorithm. On the other side, IPDs is what you have in a society where different agents are trying to cooperate / compete for resources. Clearly, super-rational agents (i.e. agents that have access to each other source code and are reflexively coherent) will act according to the same information, so no exploitation is possible, but this is an extreme case, better suited to treat problems in artificial coordination, rather than describing a real situation. Indeed some psychologists (e.g. Haidt) think that language and higher cognition evolved to serve the need of a “theory of mind” (model and influence other agents).
I recall a paper written by a student of Scott Aaronson about an IPD tournament (mentioned in the article about Eigenmorality). Indeed the winners were agents that kept a model of the opponent and responded in kind: T-f-T wasn’t by far the optimal algorithm.
On the other side, IPDs is what you have in a society where different agents are trying to cooperate / compete for resources. Clearly, super-rational agents (i.e. agents that have access to each other source code and are reflexively coherent) will act according to the same information, so no exploitation is possible, but this is an extreme case, better suited to treat problems in artificial coordination, rather than describing a real situation.
Indeed some psychologists (e.g. Haidt) think that language and higher cognition evolved to serve the need of a “theory of mind” (model and influence other agents).