At the end of paragraph 2 and the other examples, you say
This exactly mirrors the Prisoner’s Dilemma.
But it doesn’t, as you point out later in the post, because the payoff matrix isn’t D-C > C-C > D-D, as you explain, but rather C-C > D-C > C-D, because of reputational effects, which is not a prisoner’s dilemma. “Prisoner’s dilemma” is a very specific term, and you are inflating it.
evolution is also strongly motivated [...] evolution will certainly take note.
I doubt that quite strongly!
The evolutionarily dominant strategy is commonly called “Tit-for-tat”—basically, cooperate if and only if you expect your opponent to do so.
That is not tit-for-tat! Tit-for-tat is start with cooperate and then parrot the opponent’s previous move. It does not do what it “expects” the opponent to do. Furthermore, if you categorically expect your opponent to cooperate, you should defect (just like you should if you expect him to defect). You only cooperate if you expect your opponent to cooperate if he expects you to cooperate ad nauseum.
This so-called “superrationality” appears even more [...]
That is not superrationality! Superrationality achieves cooperation by reasoning that you and your opponent will get the same result for the same reasons, so you should cooperate in order to logically bind your result to C-C (since C-C and D-D are the only two options). What is with all this misuse of terminology? You write like the agents in the examples of this game are using causal decision theory (which defects all the time no matter what) and then bring up elements that cannot possibly be implemented in causal decision theory, and it grinds my gears!
And if two people with these sorts of emotional hangups play the Prisoner’s Dilemma together, they’ll end up cooperating on all hundred crimes, getting out of jail in a mere century and leaving rational utility maximizers to sit back and wonder how they did it.
This is in direct violation of one of the themes of Less Wrong. If “rational expected utility maximizers” are doing worse than “irrational emotional hangups”, then you’re using a wrong definition of “rational”. You do this throughout the post, and it’s especially jarring because you are or were one of the best writers for this website.
playing as a “rational economic agent” gets you a bad result
9_9
[...] anger makes us irrational. But this is the good kind of irrationality [...]
“The good kind of irrationality” is like “the good kind of bad thing”. An oxymoron, by definition.
[...] if we’re playing an Ultimatum Game against a human, and that human precommits to rejecting any offer less than 50-50, we’re much more likely to believe her than if we were playing against a rational utility-maximizing agent
Bullshit. A rational agent is going to do what works. We know this because we stipulated that it was rational. If you mean to say a “stupid number crunching robot that misses obvious details like how to play ultimatum games” then sure it might do as you describe. But don’t call it “rational”.
It is distasteful and a little bit contradictory to the spirit of rationality to believe it should lose out so badly to simple emotion, and the problem might be correctable.
I agree with pretty much everything you’ve said here, except:
You only cooperate if you expect your opponent to cooperate if he expects you to cooperate ad nauseum.
You don’t actually need to continue this chain—if you’re playing against any opponent which cooperates iff you cooperate, then you want to cooperate—even if the opponent would also cooperate against someone who cooperated no matter what, so your statement is also true without the “ad nauseum” (provided the opponent would defect if you defected).
You’re reading this uncharitably. There are also parts that are unclear on Yvain’s part, sure, but not to the extent that you claim.
The original group project situation Yvain explores does mirror the Prisoner’s Dilemma. Then, later, he introduces reputational effects to illustrate one of the Real World Solutions to the Prisoner’s Dilemma that we have already developed.
It’s not made crystal clear....
So one might expect the real world to have produced some practical solutions to Prisoners’ Dilemmas. One of the best known such systems is called “society”. You may have heard of it.
Well, actually it is.
...
Evolution
I understood Yvain to be speaking metaphorically, or perhaps tongue-in-cheek, when talking about what evolution would take note of. I believe this was his intention, and furthermore is a reasonable reading given our knowledge of Yvain.
This is in direct violation of one of the themes of Less Wrong. If “rational expected utility maximizers” are doing worse than “irrational emotional hangups”, then you’re using a wrong definition of “rational”. You do this throughout the post, and it’s especially jarring because you are or were one of the best writers for this website.
I expect that Yvain used ‘rational’ against the theme of LW on purpose, to create a tension—rationality failing to outperform emotional hangups is a contradiction, that would motivate readers to find the false premise or re-analyse the situation.
I do concur with your point about tit-for-tat. Similarly for super-rationality; although it’s possible Yvain is not familiar with Hofstadter’s definition and was using ‘super’ as an intensifier, it seems unlikely.
I had this downvoted based on on form and irritating tone before I looked closely and decided enough of the quotes from Yvain are, indeed, plainly wrong and I encourage hearty dismissal.
You do this throughout the post, and it’s especially jarring because you are or were one of the best writers for this website.
Agree. Who is he and what has he done to the real Yvain?
The first time I read the post, I stopped reading when “tit-for-tat” and “superrationality” were misused in two consecutive sentences. Sadly, that part seems to be still inaccurate after Yvain edited it, because TFT is not dominant in the 100-fold repeated PD, if the strategy pool contains strategies that feed on TFT.
The first time I read the post, I stopped reading when “tit-for-tat” and “superrationality” were misused in two consecutive sentences. Sadly, that part seems to be still inaccurate after Yvain edited it, because TFT is not dominant in the 100-fold repeated PD, if the strategy pool contains strategies that feed on TFT.
To be fair he doesn’t seem to make the claim that TFT is dominant in the fixed length iterated PD. (I noticed how outraged I was that Yvain was making such a basic error so I thought I should double check before agreeing emphatically!) Even so I’m not comfortable with just saying TFT is “evolutionarily dominant” in completely unspecified circumstances.
You’ll notice I used scare quotes around most of the words you objected to. I’m trying to point out the apparent paradox, using the language that game theorists and other people not already on this website would use, without claiming that the paradox is real or unsolvable.
“This so-called “superrationality” ” in the post is still wrong, I think. Would work without “so-called”, since the meaning is clear from the context, but it’s not conventional usage.
This is disturbing. I had been looking forward to the sequence, but based on this comment and some others I’m starting to wonder how accurate it is overall.
This is disturbing. I had been looking forward to the sequence, but based on this comment and some others I’m starting to wonder how accurate it is overall.
Overall, it is accurate. There are nits to pick, sure, and some loose language, but the actual math and theory is solid so far. The generalisations to real life
If I am assigned to work on a school project with a group … Diplomacy—both the concept and the board game … Civilization—again, both the concept and the game … global warming … etc
I am largely appreciative of your overall comment, but “rational” is a historically legitimate term to describe naive utility-maximisers in this manner. The original post introduced it in inverted commas, suggesting a special usage of the term. While there are less ambiguous ways this could have been expressed, it seems to me the main benefit of doing so would be to pre-empt people complaining about an unfavourable usage of the term “rational”. Your response to it seems excessive.
I want to point out that Eliezer’s (and LW’s general) use of the word ‘rationality’ is entirely different from the use of the word in the game theory literature, where it usually means VNM-rationaliity, or is used to elaborate concepts like sequential rationality in SPNE-type equilibria.
ETA: Reading Grognor’s reply to the parent, it seems that much of the negative affect is due to inconsistent use of the word ‘rational(ity)’ on LW. Maybe it’s time to try yet again to taboo LW’s ‘rationality’ to avoid the namespace collision with academic literature.
I want to point out that Eliezer’s (and LW’s general) use of the word ‘rationality’ is entirely different from the use of the word in the game theory literature
And the common usage of ‘rational’ on lesswrong should be different to what is used in a significant proportion of game theory literature. Said literature gives advice, reasoning and conclusions that is epistemically, instrumentally and normatively bad. According to the basic principles of the site it is in fact stupid and not-rational to defect against a clone of yourself in a true Prisoner’s Dilemma. A kind of stupidity that is not too much different to being ‘rational’ like Spock.
ETA: Reading Grognor’s reply to the parent, it seems that much of the negative affect is due to inconsistent use of the word ‘rational(ity)’ on LW. Maybe it’s time to try yet again to taboo LW’s ‘rationality’ to avoid the namespace collision with academic literature.
No. The themes of epistemic and instrumental rationality are the foundational premise of the site. It is right there in the tagline on the top of the page. I oppose all attempts to replace instrumental rationality with something that involves doing stupid things.
Said literature gives advice, reasoning and conclusions that is epistemically, instrumentally and normatively bad.
This is a recurring issue, so perhaps my instructor and textbooks were atypical: we never discussed or even cared whether someone should defect on PD in my game theory course. The bounds were made clear to us in lecture – game theory studies concepts like Nash equilibria and backward induction (using the term ‘rationality’ to mean VNM-rationality) and applies them to situations like PD; that is all. The use of any normative language in homework sets or exams was pretty much automatically marked incorrect. What one ‘should’ or ‘ought’ to do were instead relegated to other courses in, e.g, economics, philosophy, political science. I’d like to know from others if this is a typical experience from a game theory course (and if anyone happens to be working in the field: if this is representative of the literature).
And the common usage of ‘rational’ on lesswrong should be different to what is used in a significant proportion of game theory literature.
No. The themes of epistemic and instrumental rationality are the foundational premise of the site. It is right there in the tagline on the top of the page. I oppose all attempts to replace instrumental rationality with something that involves doing stupid things.
Upon reflection, I tend to agree with these statements. In this case, perhaps we should taboo ‘rationality’ in its game theoretic meaning – use the phrase ‘VNM-rationality’ whenever that is meant instead of LW’s ‘rationality’.
How about let’s not taboo anything, just make it clear up front what is meant when really necessary. I would prefer that because I think such taboos contribute to the entry barrier for every LW newcomer; I don’t want newcomers used to the game-theoretic jargon to keep unwittingly running afoul of this and getting downvoted.
Perhaps it would be useful if Yvain inserted a clarification of this early in the Sequence.
The use of any normative language in homework sets or exams was pretty much automatically marked incorrect.
The normative claim is one I am making now about the ‘rationality’ theories in question. It is the same kind of normative claim I make when I say “empirical tests are better than beliefs from ad baculum”.
Upon reflection, I tend to agree with these statements. In this case, perhaps we should taboo ‘rationality’ in its game theoretic meaning – use the phrase ‘VNM-rationality’ whenever that is meant instead of LW’s ‘rationality’.
I could agree to that—conditional on confirmation from one of the Vladimirs that the axioms in question do, in fact, imply the faux-rational (CDT like) conclusions the term would be used to represent. I don’t actually see it at a glance and would expect another hidden assumption to be required. I wouldn’t be comfortable using the term without confirmation.
The normative claim is one I am making now about the ‘rationality’ theories in question.
I quoted badly; I believe there was a misunderstanding. The first quote in the parent to this should be taken in the context of your sentence segment that “Said literature gives advice”. In my paragraph, I was objecting to this from my experiences in my course, where I did not receive any advice on what to do in games like PD. Instead, the type of advice that I received was on how to calculate Nash equilibria and find SPNEs.
Otherwise, I am mostly in agreement with the latter part of that sentence. (ETA: That is, I agree that if current game theoretic equilibrium solutions are taken as advice on what one ought to do, then that is often epistemically, instrumentally, and normatively bad.)
More ETA:
conditional on confirmation from one of the Vladimirs that the axioms in question do, in fact, imply the faux-rational (CDT like) conclusions the term would be used to represent. I don’t actually see it at a glance and would expect another hidden assumption to be required.
You are correct – VNM-rationality is incredibly weak (though humans don’t satisfy it). It is, after all, logically equivalent to the existence of a utility function (the proof of this by von Neumann and Morgenstern led to the eponymous VNM theorem). The faux-rationality on LW and in popular culture requires much stronger assumptions. But again, I don’t think these assumptions are made in the game theory literature – I think that faux-rationality is misattributed to game theory. The game theory I was taught used only VNM-rationality, and gave no advice.
Said literature gives advice, reasoning and conclusions that is epistemically, instrumentally and normatively bad.
I remember hearing about studies where economics and game theory students ended up less “moral” by many usual measures after completing their courses. Less inclined to co-operate, more likely to lie and cheat, more concerned about money, more likely to excuse overtly selfish behaviour and so on. And then these fine, new, upstanding citizens, went on to become the next generation of bankers, traders, stock-brokers, and advisers to politicians and industry. The rest as they say is history.
Said literature gives advice, reasoning and conclusions that is epistemically, instrumentally and normatively bad.
Said literature makes statements about what is game-theory-rational. Those statements are only epistemically, instrumentally or normatively bad if you take them to be statements about what is LW-rational or “rational” in the layperson’s sense.
Ideally we’d use different terms for game-theory-rational and LW-rational, but in the meantime we just need to keep the distinction clear in our heads so that we don’t accidentally equivocate between the two.
Said literature makes statements about what is game-theory-rational. Those statements are only epistemically, instrumentally or normatively bad if you take them to be statements about what is LW-rational or “rational” in the layperson’s sense.
Disagree on instrumentally and normatively. Agree regarding epistemically—at least when the works are careful with what claims are made. Also disagree with the “game-theory-rational”, although I understand the principle you are trying to get at. A more limited claim needs to be made or more precise terminology.
I would be interested in reading about the bases for your disagreement. Game theory is essentially the exploration of what happens if you postulate entities who are perfectly informed, personal utility-maximisers who do not care at all either way about other entities. There’s no explicit or implicit claim that people ought to behave like those entities, thus no normative content whatsoever. So I can’t see how the game theory literature could be said to give normatively bad advice, unless the speaker misunderstood the definition of rationality being used, and thought that some definition of rationality was being used in which rationality is normative.
I’m not sure what negative epistemic or instrumental outcomes you foresee either, but I’m open to the possibility that there are some.
Is there a term you prefer to “game-theory-rational” that captures the same meaning? As stated above, game theory is the exploration of what happens when entities that are “rational” by that specific definition interact with the world or each other, so it seems like the ideal term to me.
I would be interested in reading about the bases for your disagreement. Game theory is essentially the exploration of what happens if you postulate entities who are perfectly informed, personal utility-maximisers who do not care at all either way about other entities.
Under this definition you can’t claim epistemic accuracy either. In particular the ‘perfectly informed’ assumption when combined with the personal utility maximization leads to different behaviors to those described as ‘rational’. (It needs to be weakened to “perfectly informed about everything except those parts of the universe that are the other agent.)
There’s no explicit or implicit claim that people ought to behave like those entities, thus no normative content whatsoever.
This isn’t about the agents having selfish desires (in fact, they don’t even have to “not care at all about other entities”—altruism determines what the utility function is, not how to maximise it.) No, this is about shoddy claims about decision theory that are either connotatively misleading or erroneous depending on how they are framed. All those poor paperclip maximisers who read such sources and take them at face value will end up producing less paperclips than they could have if they knew the correct way to interact with the staples maximisers in contrived scenarios.
This isn’t about the agents having selfish desires (in fact, they don’t even have to “not care at all about other entities”—altruism determines what the utility function is, not how to maximise it.)
This is wrong. The standard assumption is that game-theory-rational entities are neither altruistic nor malevolent. Otherwise the Prisoner’s Dilemma wouldn’t be a dilemma in game theory. It’s only a dilemma as long as both players are solely interested in their own outcomes. As soon as you allow players to have altruistic interests in other players’ outcomes it ceases to be a dilemma.
You can do similar mathematical analyses with altruistic agents, but at that point speaking strictly you’re doing decision-theoretic calculations or possibly utilitarian calculations not game-theoretic calculations.
Utilitarian ethics, game theory and decision theory are three different things, and it seems to me your criticism assumes that statements about game theory should be taken as statements about utilitarian ethics or statements about decision theory. I think that is an instance of the fallacy of composition and we’re better served to stay very aware of the distinctions between those three frameworks.
This is wrong. The standard assumption is that game-theory-rational entities are neither altruistic nor malevolent.
No, it just isn’t. Game theory is completely agnostic about what the preferences of the players are based on. Game theory takes a payoff matrix and calculates things like Nash Equilibrium and Dominant Strategies. The verbal description of why the payoff matrix happens to be as it is is fluff.
Otherwise the Prisoner’s Dilemma wouldn’t be a dilemma in game theory. It’s only a dilemma as long as both players are solely interested in their own outcomes. As soon as you allow players to have altruistic interests in other players’ outcomes it ceases to be a dilemma.
As soon as you allow altruistic interests the game ceases to be a Prisoner’s Dilemma. The dilemma (and game theory in general) relies on the players being perfectly selfish in the sense that they ruthlessly maximise their own payoffs as they are defined, not in the sense that those payoffs must never refer to aspects of the universe that happen to include the physical state of the other agents.
Consider the Codependent Prisoner’s Dilemma. Romeo and Juliet have been captured and the guards are trying to extort confessions out of them. However Romeo and Juliet are both lovesick and infatuated and care only about what happens to their lover, not what happens to themselves. Naturally the guards offer Romeo the deal “If you confess we’ll let Juliet go and you’ll get 10 years but if you don’t confess you’ll both get 1 year” (and vice versa, with a both confess clause in there somewhere). Game theory is perfectly equipped at handling this game. In fact, so much so that it wouldn’t even bother calling it a new name. It’s just a Prisoner’s Dilemma and the fact that the conflict of interests between Romeo and Juliet happens to be based on codependent altruism rather than narcissism is outside the scope of what game theorists care about.
It seems from my perspective that we are talking past each other and that your responses are no longer tracking the original point. I don’t personally think that deserves upvotes, but others obviously differ.
Your original claim was that:
Said literature gives advice, reasoning and conclusions that is epistemically, instrumentally and normatively bad.
Now given that game theory is not making any normative claims, it can’t be saying things which are normatively bad. Similarly since game theory does not say that you should either go out and act like a game-theory-rational agent or that you should act as if others will do so, it can’t be saying anything instrumentally bad either.
I just don’t see how it could even be possible for game theory to do what you claim it does. That would be like stating that a document describing the rules of poker was instrumentally and normatively bad because it encouraged wasteful, zero-sum gaming. It would be mistaking description for prescription.
We have already agreed, I think, that there is nothing epistemically bad about game theory taken as it is.
Everything below responds to the off-track discussion above and can be safely ignored by posters not specifically interested in that digression.
In game theory each player’s payoff matrix is their own. Notice that Codependent Romeo does not care where Codependent Juliet ends up in her payoff matrix. If Codependent Romeo was altruistic in the sense of wanting to maximise Juliet’s satisfaction with her payoff, he’d be keeping silent. Because Codependent Romeo is game-theory-rational, he’s indifferent to Codependent Juliet’s satisfaction with her outcome and only cares about maximising his personal payoff.
The standard assumption in a game-theoretic analysis is that a poker player wants money, a chess player wants to win chess games and so on, and that they are indifferent to their opponent(s) opinion about the outcome, just as Codependent Romeo is maximising his own payoff matrix and is indifferent to Codependent Juliet’s.
That is what we attempt to convey when we tell people that game-theory-rational players are neither benevolent nor malevolent. Even if you incorporate something you want to call “altruism” into their preference order, they still don’t care directly about where anyone else ends up in those other peoples’ preference orders.
Now given that game theory is not making any normative claims, it can’t be saying things which are normatively bad.
Not true. The word ‘connotations’ comes to mind. As does “reframing to the extent of outright redefining a critical keyword”. That is not a normatively neutral act. It is legitimate for me to judge it and I choose to do so—negatively.
I think this was a legitimate use of “by definition”, since it’s the definition we use on this website. You’re right that “rational” has often meant “blindly crunching numbers without looking at all available information &c.” but I thought we had a widespread agreement here not to use the word like that.
You’re right that my response seems excessive, but I don’t know if it actually is excessive rather than merely seeming so.
I think this was a legitimate use of “by definition”, since it’s the definition we use on this website.
A term “bad rationality” is also used on this website. It is a partial rationality, and it may be harmful. On the other hand, as humans, partial rationality is all we have, don’t we?
But now I am discussing labels on the map, not the territory.
You’re attaching a negative connotation where there doesn’t have to be one. In econ and game theory literature, “rational” means something else, not necessarily something bad. It also refers to something specific. If we want to talk about that specific referent, we have limited options.
I would propose suffixing alternative uses of the word “rational” with a disambiguating particle. Thus above, Yvain could have used “econ-rational”. If we ever have cause to talk about the Rationalist philosophical tradition, they can be “p-Rationalists”. Annoyingly, I don’t actually believe we need to do this for disambiguation purposes.
Upvoted. By the way, I think you meant to type “C-C > D-C > D-D” instead of “C-C > D-C > D-C”, and you might also want to include C-D. Not sure whether C-D is supposed to be last or second last in Yvain’s example, because of the reputational effects.
There are many problems here.
At the end of paragraph 2 and the other examples, you say
But it doesn’t, as you point out later in the post, because the payoff matrix isn’t D-C > C-C > D-D, as you explain, but rather C-C > D-C > C-D, because of reputational effects, which is not a prisoner’s dilemma. “Prisoner’s dilemma” is a very specific term, and you are inflating it.
I doubt that quite strongly!
That is not tit-for-tat! Tit-for-tat is start with cooperate and then parrot the opponent’s previous move. It does not do what it “expects” the opponent to do. Furthermore, if you categorically expect your opponent to cooperate, you should defect (just like you should if you expect him to defect). You only cooperate if you expect your opponent to cooperate if he expects you to cooperate ad nauseum.
That is not superrationality! Superrationality achieves cooperation by reasoning that you and your opponent will get the same result for the same reasons, so you should cooperate in order to logically bind your result to C-C (since C-C and D-D are the only two options). What is with all this misuse of terminology? You write like the agents in the examples of this game are using causal decision theory (which defects all the time no matter what) and then bring up elements that cannot possibly be implemented in causal decision theory, and it grinds my gears!
This is in direct violation of one of the themes of Less Wrong. If “rational expected utility maximizers” are doing worse than “irrational emotional hangups”, then you’re using a wrong definition of “rational”. You do this throughout the post, and it’s especially jarring because you are or were one of the best writers for this website.
9_9
“The good kind of irrationality” is like “the good kind of bad thing”. An oxymoron, by definition.
Bullshit. A rational agent is going to do what works. We know this because we stipulated that it was rational. If you mean to say a “stupid number crunching robot that misses obvious details like how to play ultimatum games” then sure it might do as you describe. But don’t call it “rational”.
You think?
Downvoted.
I agree with pretty much everything you’ve said here, except:
You don’t actually need to continue this chain—if you’re playing against any opponent which cooperates iff you cooperate, then you want to cooperate—even if the opponent would also cooperate against someone who cooperated no matter what, so your statement is also true without the “ad nauseum” (provided the opponent would defect if you defected).
You’re right. I assumed symmetry, which was wrong.
You’re reading this uncharitably. There are also parts that are unclear on Yvain’s part, sure, but not to the extent that you claim.
The original group project situation Yvain explores does mirror the Prisoner’s Dilemma. Then, later, he introduces reputational effects to illustrate one of the Real World Solutions to the Prisoner’s Dilemma that we have already developed.
It’s not made crystal clear....
Well, actually it is.
...
I understood Yvain to be speaking metaphorically, or perhaps tongue-in-cheek, when talking about what evolution would take note of. I believe this was his intention, and furthermore is a reasonable reading given our knowledge of Yvain.
I expect that Yvain used ‘rational’ against the theme of LW on purpose, to create a tension—rationality failing to outperform emotional hangups is a contradiction, that would motivate readers to find the false premise or re-analyse the situation.
I do concur with your point about tit-for-tat. Similarly for super-rationality; although it’s possible Yvain is not familiar with Hofstadter’s definition and was using ‘super’ as an intensifier, it seems unlikely.
I had this downvoted based on on form and irritating tone before I looked closely and decided enough of the quotes from Yvain are, indeed, plainly wrong and I encourage hearty dismissal.
Agree. Who is he and what has he done to the real Yvain?
Great critique!
The first time I read the post, I stopped reading when “tit-for-tat” and “superrationality” were misused in two consecutive sentences. Sadly, that part seems to be still inaccurate after Yvain edited it, because TFT is not dominant in the 100-fold repeated PD, if the strategy pool contains strategies that feed on TFT.
To be fair he doesn’t seem to make the claim that TFT is dominant in the fixed length iterated PD. (I noticed how outraged I was that Yvain was making such a basic error so I thought I should double check before agreeing emphatically!) Even so I’m not comfortable with just saying TFT is “evolutionarily dominant” in completely unspecified circumstances.
You’ll notice I used scare quotes around most of the words you objected to. I’m trying to point out the apparent paradox, using the language that game theorists and other people not already on this website would use, without claiming that the paradox is real or unsolvable.
“This so-called “superrationality” ” in the post is still wrong, I think. Would work without “so-called”, since the meaning is clear from the context, but it’s not conventional usage.
This is disturbing. I had been looking forward to the sequence, but based on this comment and some others I’m starting to wonder how accurate it is overall.
Overall, it is accurate. There are nits to pick, sure, and some loose language, but the actual math and theory is solid so far. The generalisations to real life
are correct as well.
By definition?
I am largely appreciative of your overall comment, but “rational” is a historically legitimate term to describe naive utility-maximisers in this manner. The original post introduced it in inverted commas, suggesting a special usage of the term. While there are less ambiguous ways this could have been expressed, it seems to me the main benefit of doing so would be to pre-empt people complaining about an unfavourable usage of the term “rational”. Your response to it seems excessive.
Agreed.
I want to point out that Eliezer’s (and LW’s general) use of the word ‘rationality’ is entirely different from the use of the word in the game theory literature, where it usually means VNM-rationaliity, or is used to elaborate concepts like sequential rationality in SPNE-type equilibria.
ETA: Reading Grognor’s reply to the parent, it seems that much of the negative affect is due to inconsistent use of the word ‘rational(ity)’ on LW. Maybe it’s time to try yet again to taboo LW’s ‘rationality’ to avoid the namespace collision with academic literature.
And the common usage of ‘rational’ on lesswrong should be different to what is used in a significant proportion of game theory literature. Said literature gives advice, reasoning and conclusions that is epistemically, instrumentally and normatively bad. According to the basic principles of the site it is in fact stupid and not-rational to defect against a clone of yourself in a true Prisoner’s Dilemma. A kind of stupidity that is not too much different to being ‘rational’ like Spock.
No. The themes of epistemic and instrumental rationality are the foundational premise of the site. It is right there in the tagline on the top of the page. I oppose all attempts to replace instrumental rationality with something that involves doing stupid things.
I do endorse avoiding excessive use of the word.
This is a recurring issue, so perhaps my instructor and textbooks were atypical: we never discussed or even cared whether someone should defect on PD in my game theory course. The bounds were made clear to us in lecture – game theory studies concepts like Nash equilibria and backward induction (using the term ‘rationality’ to mean VNM-rationality) and applies them to situations like PD; that is all. The use of any normative language in homework sets or exams was pretty much automatically marked incorrect. What one ‘should’ or ‘ought’ to do were instead relegated to other courses in, e.g, economics, philosophy, political science. I’d like to know from others if this is a typical experience from a game theory course (and if anyone happens to be working in the field: if this is representative of the literature).
Upon reflection, I tend to agree with these statements. In this case, perhaps we should taboo ‘rationality’ in its game theoretic meaning – use the phrase ‘VNM-rationality’ whenever that is meant instead of LW’s ‘rationality’.
How about let’s not taboo anything, just make it clear up front what is meant when really necessary. I would prefer that because I think such taboos contribute to the entry barrier for every LW newcomer; I don’t want newcomers used to the game-theoretic jargon to keep unwittingly running afoul of this and getting downvoted.
Perhaps it would be useful if Yvain inserted a clarification of this early in the Sequence.
The normative claim is one I am making now about the ‘rationality’ theories in question. It is the same kind of normative claim I make when I say “empirical tests are better than beliefs from ad baculum”.
I could agree to that—conditional on confirmation from one of the Vladimirs that the axioms in question do, in fact, imply the faux-rational (CDT like) conclusions the term would be used to represent. I don’t actually see it at a glance and would expect another hidden assumption to be required. I wouldn’t be comfortable using the term without confirmation.
I quoted badly; I believe there was a misunderstanding. The first quote in the parent to this should be taken in the context of your sentence segment that “Said literature gives advice”. In my paragraph, I was objecting to this from my experiences in my course, where I did not receive any advice on what to do in games like PD. Instead, the type of advice that I received was on how to calculate Nash equilibria and find SPNEs.
Otherwise, I am mostly in agreement with the latter part of that sentence. (ETA: That is, I agree that if current game theoretic equilibrium solutions are taken as advice on what one ought to do, then that is often epistemically, instrumentally, and normatively bad.)
More ETA:
You are correct – VNM-rationality is incredibly weak (though humans don’t satisfy it). It is, after all, logically equivalent to the existence of a utility function (the proof of this by von Neumann and Morgenstern led to the eponymous VNM theorem). The faux-rationality on LW and in popular culture requires much stronger assumptions. But again, I don’t think these assumptions are made in the game theory literature – I think that faux-rationality is misattributed to game theory. The game theory I was taught used only VNM-rationality, and gave no advice.
I remember hearing about studies where economics and game theory students ended up less “moral” by many usual measures after completing their courses. Less inclined to co-operate, more likely to lie and cheat, more concerned about money, more likely to excuse overtly selfish behaviour and so on. And then these fine, new, upstanding citizens, went on to become the next generation of bankers, traders, stock-brokers, and advisers to politicians and industry. The rest as they say is history.
Said literature makes statements about what is game-theory-rational. Those statements are only epistemically, instrumentally or normatively bad if you take them to be statements about what is LW-rational or “rational” in the layperson’s sense.
Ideally we’d use different terms for game-theory-rational and LW-rational, but in the meantime we just need to keep the distinction clear in our heads so that we don’t accidentally equivocate between the two.
Disagree on instrumentally and normatively. Agree regarding epistemically—at least when the works are careful with what claims are made. Also disagree with the “game-theory-rational”, although I understand the principle you are trying to get at. A more limited claim needs to be made or more precise terminology.
I would be interested in reading about the bases for your disagreement. Game theory is essentially the exploration of what happens if you postulate entities who are perfectly informed, personal utility-maximisers who do not care at all either way about other entities. There’s no explicit or implicit claim that people ought to behave like those entities, thus no normative content whatsoever. So I can’t see how the game theory literature could be said to give normatively bad advice, unless the speaker misunderstood the definition of rationality being used, and thought that some definition of rationality was being used in which rationality is normative.
I’m not sure what negative epistemic or instrumental outcomes you foresee either, but I’m open to the possibility that there are some.
Is there a term you prefer to “game-theory-rational” that captures the same meaning? As stated above, game theory is the exploration of what happens when entities that are “rational” by that specific definition interact with the world or each other, so it seems like the ideal term to me.
Under this definition you can’t claim epistemic accuracy either. In particular the ‘perfectly informed’ assumption when combined with the personal utility maximization leads to different behaviors to those described as ‘rational’. (It needs to be weakened to “perfectly informed about everything except those parts of the universe that are the other agent.)
This isn’t about the agents having selfish desires (in fact, they don’t even have to “not care at all about other entities”—altruism determines what the utility function is, not how to maximise it.) No, this is about shoddy claims about decision theory that are either connotatively misleading or erroneous depending on how they are framed. All those poor paperclip maximisers who read such sources and take them at face value will end up producing less paperclips than they could have if they knew the correct way to interact with the staples maximisers in contrived scenarios.
This is wrong. The standard assumption is that game-theory-rational entities are neither altruistic nor malevolent. Otherwise the Prisoner’s Dilemma wouldn’t be a dilemma in game theory. It’s only a dilemma as long as both players are solely interested in their own outcomes. As soon as you allow players to have altruistic interests in other players’ outcomes it ceases to be a dilemma.
You can do similar mathematical analyses with altruistic agents, but at that point speaking strictly you’re doing decision-theoretic calculations or possibly utilitarian calculations not game-theoretic calculations.
Utilitarian ethics, game theory and decision theory are three different things, and it seems to me your criticism assumes that statements about game theory should be taken as statements about utilitarian ethics or statements about decision theory. I think that is an instance of the fallacy of composition and we’re better served to stay very aware of the distinctions between those three frameworks.
No, it just isn’t. Game theory is completely agnostic about what the preferences of the players are based on. Game theory takes a payoff matrix and calculates things like Nash Equilibrium and Dominant Strategies. The verbal description of why the payoff matrix happens to be as it is is fluff.
As soon as you allow altruistic interests the game ceases to be a Prisoner’s Dilemma. The dilemma (and game theory in general) relies on the players being perfectly selfish in the sense that they ruthlessly maximise their own payoffs as they are defined, not in the sense that those payoffs must never refer to aspects of the universe that happen to include the physical state of the other agents.
Consider the Codependent Prisoner’s Dilemma. Romeo and Juliet have been captured and the guards are trying to extort confessions out of them. However Romeo and Juliet are both lovesick and infatuated and care only about what happens to their lover, not what happens to themselves. Naturally the guards offer Romeo the deal “If you confess we’ll let Juliet go and you’ll get 10 years but if you don’t confess you’ll both get 1 year” (and vice versa, with a both confess clause in there somewhere). Game theory is perfectly equipped at handling this game. In fact, so much so that it wouldn’t even bother calling it a new name. It’s just a Prisoner’s Dilemma and the fact that the conflict of interests between Romeo and Juliet happens to be based on codependent altruism rather than narcissism is outside the scope of what game theorists care about.
It seems from my perspective that we are talking past each other and that your responses are no longer tracking the original point. I don’t personally think that deserves upvotes, but others obviously differ.
Your original claim was that:
Now given that game theory is not making any normative claims, it can’t be saying things which are normatively bad. Similarly since game theory does not say that you should either go out and act like a game-theory-rational agent or that you should act as if others will do so, it can’t be saying anything instrumentally bad either.
I just don’t see how it could even be possible for game theory to do what you claim it does. That would be like stating that a document describing the rules of poker was instrumentally and normatively bad because it encouraged wasteful, zero-sum gaming. It would be mistaking description for prescription.
We have already agreed, I think, that there is nothing epistemically bad about game theory taken as it is.
Everything below responds to the off-track discussion above and can be safely ignored by posters not specifically interested in that digression.
In game theory each player’s payoff matrix is their own. Notice that Codependent Romeo does not care where Codependent Juliet ends up in her payoff matrix. If Codependent Romeo was altruistic in the sense of wanting to maximise Juliet’s satisfaction with her payoff, he’d be keeping silent. Because Codependent Romeo is game-theory-rational, he’s indifferent to Codependent Juliet’s satisfaction with her outcome and only cares about maximising his personal payoff.
The standard assumption in a game-theoretic analysis is that a poker player wants money, a chess player wants to win chess games and so on, and that they are indifferent to their opponent(s) opinion about the outcome, just as Codependent Romeo is maximising his own payoff matrix and is indifferent to Codependent Juliet’s.
That is what we attempt to convey when we tell people that game-theory-rational players are neither benevolent nor malevolent. Even if you incorporate something you want to call “altruism” into their preference order, they still don’t care directly about where anyone else ends up in those other peoples’ preference orders.
Not true. The word ‘connotations’ comes to mind. As does “reframing to the extent of outright redefining a critical keyword”. That is not a normatively neutral act. It is legitimate for me to judge it and I choose to do so—negatively.
I think this was a legitimate use of “by definition”, since it’s the definition we use on this website. You’re right that “rational” has often meant “blindly crunching numbers without looking at all available information &c.” but I thought we had a widespread agreement here not to use the word like that.
You’re right that my response seems excessive, but I don’t know if it actually is excessive rather than merely seeming so.
A term “bad rationality” is also used on this website. It is a partial rationality, and it may be harmful. On the other hand, as humans, partial rationality is all we have, don’t we?
But now I am discussing labels on the map, not the territory.
You’re attaching a negative connotation where there doesn’t have to be one. In econ and game theory literature, “rational” means something else, not necessarily something bad. It also refers to something specific. If we want to talk about that specific referent, we have limited options.
I would propose suffixing alternative uses of the word “rational” with a disambiguating particle. Thus above, Yvain could have used “econ-rational”. If we ever have cause to talk about the Rationalist philosophical tradition, they can be “p-Rationalists”. Annoyingly, I don’t actually believe we need to do this for disambiguation purposes.
Upvoted. By the way, I think you meant to type “C-C > D-C > D-D” instead of “C-C > D-C > D-C”, and you might also want to include C-D. Not sure whether C-D is supposed to be last or second last in Yvain’s example, because of the reputational effects.