the only strategies for the True Prisoner’s Dilemma are CooperateBot and DefectBot
I disagree. The Prisoner’s Dilemma does not specify that you are blind as to the nature of your opponent. “Visible source code” is a device to allow bots an analog of the many character analysis tools available when humans play against humans.
If you think you’re playing against Omega, or if you use TDT and you think you’re playing against someone else who uses TDT, then you should cooperate. I don’t think an inability to reason about your opponent makes the game more “True”.
The underlying insight there is “understand which game you are playing”
This is one of the underlying insights, but another is “your monkey brain may be programmed to act optimally under strange parameters”. Someone else linked a post by MBlume which makes a similar point (in, perhaps, a less aggravating manner).
It may be that you gain access to certain resources only when you believe things you epistemically shouldn’t. In such cases, cultivating false beliefs (preferably compartmentalized) can be very useful.
I apologize for the aggravation. My aim was to be provocative and perhaps uncomfortable, but not aggravating.
I disagree. The Prisoner’s Dilemma does not specify that you are blind as to the nature of your opponent.
The transparent version of the Prisoner’s Dilemma, and the more complicated ‘shared source code’ version that shows up on LW, are generally considered variants of the basic PD.
In contrast to games where you can say things like “I cooperate if they cooperate, and I defect if they defect,” in the basic game you either say “I cooperate” or “I defect.” Now, you might know some things about them, and they might know some things about you, but there’s no causal connection between your action and their action, like there is if they’re informed of your action, they’re informed of your source code, or they have the ability to perceive the future.
I apologize for the aggravation.
“Aggravating” may have been too strong a word; “disappointed” might have been better, that I saw content I mostly agreed with presented in a way I mostly disagreed with, with the extra implication that the presentation was possibly more important than the content.
To me, a “vanilla” Prisoner’s Dilemma involves actual human prisoners who may reason about their partners. I don’t mean to imply that I think “standard” PD involves credible pre-commitments nor perfect knowledge of the opponent. While I agree that in standard PD there’s no causal connection between actions, there can be logical connections between actions that make for interesting strategies (eg if you expect them to use TDT).
On this point, I’m inclined to think that we agree and are debating terminology.
“Aggravating” may have been too strong a word; “disappointed” might have been better
That’s even worse! :-)
I readily admit that my presentation is tailored to my personality, and I understand how others may find it grating.
That said, a secondary goal of this post was to instill doubt in concepts that look sacred (terminal goals, epistemic rationality) and encourage people to consider that even these may be sacrificed for instrumental gains.
It seems you already grasp the tradeoffs between epistemic and instrumental rationality and that you can consistently reach mental states that are elusive to naive epistemically rational agents, and that you’ve come to these conclusions by a different means than I. By my analysis, there are many others who need a push before they are willing to even consider “terminal goals” and “false beliefs” as strategic tools. This post caters more to them.
I’d be very interested to hear more about how you’ve achieved similar results with different techniques!
If you think you’re playing against Omega, or if you use TDT and you think you’re playing against someone else who uses TDT, then you should cooperate.
Although strictly speaking, if you’re playing against Omega or another TDT user, you’re not really playing a true PD any more. (Why? Because mutual cooperation reveals preferences that are inconsistent with the preferences implied by a true PD payoff table.)
Reading the ensuing disagreement, this seems like a good occasion to ask whether this is a policy suggestion, and if so what it is. I don’t think So8res disagrees about any theorems (e.g. about dominant strategies) over formalisms of game theory/PD, so it seems like the scope of the disagreement is (at most) pretty much how one should use the phrase ‘Prisoner’s Dilemma’, and that there are more direct ways of arguing that point, e.g. pointing to ways in which the term (‘Prisoner’s Dilemma’) originally used for the formal PD also being used for e.g. playing against various bots causes thought to systematically go astray/causes confusion/etc.
Reading the ensuing disagreement, this seems like a good occasion to ask whether this is a policy suggestion, and if so what it is. [...] it seems like the scope of the disagreement is (at most) pretty much how one should use the phrase ‘Prisoner’s Dilemma’
Pretty much. Cashing out my disagreement as a policy recommendation: don’t call a situation a true PD if that situation’s feasible set doesn’t include (C, D) & (D, C). Otherwise one might deduce that cooperation is the rational outcome for the one-shot, vanilla PD. It isn’t, even if believing it is puts one in goodcompany.
As I understand it, Hofstadter’s advocacy of cooperation was limited to games with some sense of source-code sharing. Basically, both agents were able to assume their co-players had an identical method of deciding on the optimal move, and that that method was optimal. That assumption allows a rather bizarre little proof that cooperation is the result said method arrives at.
And think about it, how could a mathematician actually advocate cooperation in pure, zero knowledge vanilla PD? That just doesn’t make any sense as a model of an intelligent human being’s opinions.
Agreed. But here is what I think Hofstadter was saying: The assumption that is used can be weaker than the assumption that the two players have an identical method. Rather, it just needs to be that they are both “smart”. And this is almost as strong a result as the true zero knowledge scenario, because most agents will do their best to be smart.
Why is he saying that “smart” agents will cooperate? Because they know that the other agent is the same as them in that respect. (In being smart, and also in knowing what being smart means.)
Now, there are some obvious holes in this, but it does hold a certain grain of truth, and is a fairly powerful result in any case. (TDT is, in a sense, a generalization of exactly this idea.)
Have you seen this explored in mathematical language? Cause it’s all so weird that there’s no way I can agree with Hofstadter to that extent. As yet, I don’t know really know what “smart” means.
Yeah, I agree, it is weird. And I think that Hofstadter is wrong: With such a vague definition of being “smart”, his conjecture fails to hold. (This is what you were saying: It’s rather vague and undefined.)
That said, TDT is an attempt to put a similar idea on firmer ground. In that sense, the TDT paper is the exploration in mathematical language of this idea that you are asking for. It isn’t Hofstadterian superrationality, but it is inspired by it, and TDT puts these amorphous concepts that Hofstadter never bothered solidifying into a concrete form.
What ygert said. So-called superrationality has a grain of truth but there are obvious holes in it (at least as originally described by Hofstadter).
And think about it, how could a mathematician actually advocate cooperation in pure, zero knowledge vanilla PD? That just doesn’t make any sense as a model of an intelligent human being’s opinions.
Sadly, even intelligent human beings have been known to believe incorrect things for bad reasons.
More to the point, I’m not accusing Hofstadter of advocating cooperation in a zero knowledge PD. I’m accusing him of advocating cooperation in a one-shot PD where both players are known to be rational. In this scenario, too, both players defect.
Hofstadter can deny this only by playing games(!) with the word “rational”. He first defines it to mean that a rational player gets the same answer as another rational player, so he can eliminate (C, D) & (D, C), and then and only then does he decide that it also means players don’t choose a dominated strategy, which eliminates (D, D). But this is silly; the avoids-dominated-strategies definition renders the gets-the-same-answer-as-another-rational-player definition superfluous (in this specific case). Suppose it had never occurred to us to use the former definition of “rational”, and we simply applied the latter definition. We’d immediately notice that neither player cooperates, because cooperation is strictly dominated according to the true PD payoff matrix, and we’d immediately eliminate all outcomes but (D, D). Hofstadter dodges this conclusion by using a gimmick to avoid consistently applying the requirement that rational players don’t leave free utility on the table.
I disagree. Mutual cooperation need not require preferences inconsistent with the payoff table: it could well be that I would prefer to defect, but I have reason to believe that my opponent’s move will be the same as my own (not via any causal mechanism but via a logical relation, eg if I know they use TDT).
Mutual cooperation need not require preferences inconsistent with the payoff table: it could well be that I would prefer to defect, but I have reason to believe that my opponent’s move will be the same as my own
But the true PD payoff table directly implies, by elimination of dominated strategies, that both players prefer D to C. If a player instead plays C, the alleged payoff table can’t have represented the actual utility the players assigned to different outcomes.
Playing against a TDT user or Omega or a copy of oneself doesn’t so much refute this conclusion as circumvent it. When both players know their opponent’s move must match their own, half of the payoff table’s entries are effectively deleted because the (C, D) & (D, C) outcomes become unreachable, which means the table’s no longer a true PD payoff table (which has four entries). And notice that with only the (C, C) & (D, D) entries left in the table, the (C, C) entry strictly dominates (D, D) for both players; both players now really & truly prefer to cooperate, and there is no lingering preference to defect.
I don’t think it’s using a non-causal decision theory that makes the difference. I could play myself in front of a mirror — so that my so-called opponent’s move is directly caused by mine, and I can think things through in purely causal terms — and my point about the payoff table having only two entries would still apply.
What makes the difference is whether non-game-theoretic considerations circumscribe the feasible set of possible outcomes before the players try to optimize within the feasible set. If I know nothing about my opponent, my feasible set has four outcomes. If my opponent is my mirror image (or a fellow TDT user, or Omega), I know my feasible set has two outcomes, because (C, D) & (D, C) are blocked a priori by the setup of the situation. If two human game theorists face off, they also end up ruling out (C, D) & (D, C), but only in the process of whittling the original feasible set of four possibilities down to the Nash equilibrium.
Thanks for the feedback!
I disagree. The Prisoner’s Dilemma does not specify that you are blind as to the nature of your opponent. “Visible source code” is a device to allow bots an analog of the many character analysis tools available when humans play against humans.
If you think you’re playing against Omega, or if you use TDT and you think you’re playing against someone else who uses TDT, then you should cooperate. I don’t think an inability to reason about your opponent makes the game more “True”.
This is one of the underlying insights, but another is “your monkey brain may be programmed to act optimally under strange parameters”. Someone else linked a post by MBlume which makes a similar point (in, perhaps, a less aggravating manner).
It may be that you gain access to certain resources only when you believe things you epistemically shouldn’t. In such cases, cultivating false beliefs (preferably compartmentalized) can be very useful.
I apologize for the aggravation. My aim was to be provocative and perhaps uncomfortable, but not aggravating.
The transparent version of the Prisoner’s Dilemma, and the more complicated ‘shared source code’ version that shows up on LW, are generally considered variants of the basic PD.
In contrast to games where you can say things like “I cooperate if they cooperate, and I defect if they defect,” in the basic game you either say “I cooperate” or “I defect.” Now, you might know some things about them, and they might know some things about you, but there’s no causal connection between your action and their action, like there is if they’re informed of your action, they’re informed of your source code, or they have the ability to perceive the future.
“Aggravating” may have been too strong a word; “disappointed” might have been better, that I saw content I mostly agreed with presented in a way I mostly disagreed with, with the extra implication that the presentation was possibly more important than the content.
To me, a “vanilla” Prisoner’s Dilemma involves actual human prisoners who may reason about their partners. I don’t mean to imply that I think “standard” PD involves credible pre-commitments nor perfect knowledge of the opponent. While I agree that in standard PD there’s no causal connection between actions, there can be logical connections between actions that make for interesting strategies (eg if you expect them to use TDT).
On this point, I’m inclined to think that we agree and are debating terminology.
That’s even worse! :-)
I readily admit that my presentation is tailored to my personality, and I understand how others may find it grating.
That said, a secondary goal of this post was to instill doubt in concepts that look sacred (terminal goals, epistemic rationality) and encourage people to consider that even these may be sacrificed for instrumental gains.
It seems you already grasp the tradeoffs between epistemic and instrumental rationality and that you can consistently reach mental states that are elusive to naive epistemically rational agents, and that you’ve come to these conclusions by a different means than I. By my analysis, there are many others who need a push before they are willing to even consider “terminal goals” and “false beliefs” as strategic tools. This post caters more to them.
I’d be very interested to hear more about how you’ve achieved similar results with different techniques!
Although strictly speaking, if you’re playing against Omega or another TDT user, you’re not really playing a true PD any more. (Why? Because mutual cooperation reveals preferences that are inconsistent with the preferences implied by a true PD payoff table.)
Reading the ensuing disagreement, this seems like a good occasion to ask whether this is a policy suggestion, and if so what it is. I don’t think So8res disagrees about any theorems (e.g. about dominant strategies) over formalisms of game theory/PD, so it seems like the scope of the disagreement is (at most) pretty much how one should use the phrase ‘Prisoner’s Dilemma’, and that there are more direct ways of arguing that point, e.g. pointing to ways in which the term (‘Prisoner’s Dilemma’) originally used for the formal PD also being used for e.g. playing against various bots causes thought to systematically go astray/causes confusion/etc.
Pretty much. Cashing out my disagreement as a policy recommendation: don’t call a situation a true PD if that situation’s feasible set doesn’t include (C, D) & (D, C). Otherwise one might deduce that cooperation is the rational outcome for the one-shot, vanilla PD. It isn’t, even if believing it is puts one in good company.
As I understand it, Hofstadter’s advocacy of cooperation was limited to games with some sense of source-code sharing. Basically, both agents were able to assume their co-players had an identical method of deciding on the optimal move, and that that method was optimal. That assumption allows a rather bizarre little proof that cooperation is the result said method arrives at.
And think about it, how could a mathematician actually advocate cooperation in pure, zero knowledge vanilla PD? That just doesn’t make any sense as a model of an intelligent human being’s opinions.
Agreed. But here is what I think Hofstadter was saying: The assumption that is used can be weaker than the assumption that the two players have an identical method. Rather, it just needs to be that they are both “smart”. And this is almost as strong a result as the true zero knowledge scenario, because most agents will do their best to be smart.
Why is he saying that “smart” agents will cooperate? Because they know that the other agent is the same as them in that respect. (In being smart, and also in knowing what being smart means.)
Now, there are some obvious holes in this, but it does hold a certain grain of truth, and is a fairly powerful result in any case. (TDT is, in a sense, a generalization of exactly this idea.)
Have you seen this explored in mathematical language? Cause it’s all so weird that there’s no way I can agree with Hofstadter to that extent. As yet, I don’t know really know what “smart” means.
Yeah, I agree, it is weird. And I think that Hofstadter is wrong: With such a vague definition of being “smart”, his conjecture fails to hold. (This is what you were saying: It’s rather vague and undefined.)
That said, TDT is an attempt to put a similar idea on firmer ground. In that sense, the TDT paper is the exploration in mathematical language of this idea that you are asking for. It isn’t Hofstadterian superrationality, but it is inspired by it, and TDT puts these amorphous concepts that Hofstadter never bothered solidifying into a concrete form.
What ygert said. So-called superrationality has a grain of truth but there are obvious holes in it (at least as originally described by Hofstadter).
Sadly, even intelligent human beings have been known to believe incorrect things for bad reasons.
More to the point, I’m not accusing Hofstadter of advocating cooperation in a zero knowledge PD. I’m accusing him of advocating cooperation in a one-shot PD where both players are known to be rational. In this scenario, too, both players defect.
Hofstadter can deny this only by playing games(!) with the word “rational”. He first defines it to mean that a rational player gets the same answer as another rational player, so he can eliminate (C, D) & (D, C), and then and only then does he decide that it also means players don’t choose a dominated strategy, which eliminates (D, D). But this is silly; the avoids-dominated-strategies definition renders the gets-the-same-answer-as-another-rational-player definition superfluous (in this specific case). Suppose it had never occurred to us to use the former definition of “rational”, and we simply applied the latter definition. We’d immediately notice that neither player cooperates, because cooperation is strictly dominated according to the true PD payoff matrix, and we’d immediately eliminate all outcomes but (D, D). Hofstadter dodges this conclusion by using a gimmick to avoid consistently applying the requirement that rational players don’t leave free utility on the table.
I disagree. Mutual cooperation need not require preferences inconsistent with the payoff table: it could well be that I would prefer to defect, but I have reason to believe that my opponent’s move will be the same as my own (not via any causal mechanism but via a logical relation, eg if I know they use TDT).
But the true PD payoff table directly implies, by elimination of dominated strategies, that both players prefer D to C. If a player instead plays C, the alleged payoff table can’t have represented the actual utility the players assigned to different outcomes.
Playing against a TDT user or Omega or a copy of oneself doesn’t so much refute this conclusion as circumvent it. When both players know their opponent’s move must match their own, half of the payoff table’s entries are effectively deleted because the (C, D) & (D, C) outcomes become unreachable, which means the table’s no longer a true PD payoff table (which has four entries). And notice that with only the (C, C) & (D, D) entries left in the table, the (C, C) entry strictly dominates (D, D) for both players; both players now really & truly prefer to cooperate, and there is no lingering preference to defect.
TDT players achieving mutual cooperation are playing the same game as causal players in Nash equilibrium.
I’m not sure how you think that the game is different when the players are using non-causal decision theories.
I don’t think it’s using a non-causal decision theory that makes the difference. I could play myself in front of a mirror — so that my so-called opponent’s move is directly caused by mine, and I can think things through in purely causal terms — and my point about the payoff table having only two entries would still apply.
What makes the difference is whether non-game-theoretic considerations circumscribe the feasible set of possible outcomes before the players try to optimize within the feasible set. If I know nothing about my opponent, my feasible set has four outcomes. If my opponent is my mirror image (or a fellow TDT user, or Omega), I know my feasible set has two outcomes, because (C, D) & (D, C) are blocked a priori by the setup of the situation. If two human game theorists face off, they also end up ruling out (C, D) & (D, C), but only in the process of whittling the original feasible set of four possibilities down to the Nash equilibrium.