I want to point out that Eliezer’s (and LW’s general) use of the word ‘rationality’ is entirely different from the use of the word in the game theory literature, where it usually means VNM-rationaliity, or is used to elaborate concepts like sequential rationality in SPNE-type equilibria.
ETA: Reading Grognor’s reply to the parent, it seems that much of the negative affect is due to inconsistent use of the word ‘rational(ity)’ on LW. Maybe it’s time to try yet again to taboo LW’s ‘rationality’ to avoid the namespace collision with academic literature.
I want to point out that Eliezer’s (and LW’s general) use of the word ‘rationality’ is entirely different from the use of the word in the game theory literature
And the common usage of ‘rational’ on lesswrong should be different to what is used in a significant proportion of game theory literature. Said literature gives advice, reasoning and conclusions that is epistemically, instrumentally and normatively bad. According to the basic principles of the site it is in fact stupid and not-rational to defect against a clone of yourself in a true Prisoner’s Dilemma. A kind of stupidity that is not too much different to being ‘rational’ like Spock.
ETA: Reading Grognor’s reply to the parent, it seems that much of the negative affect is due to inconsistent use of the word ‘rational(ity)’ on LW. Maybe it’s time to try yet again to taboo LW’s ‘rationality’ to avoid the namespace collision with academic literature.
No. The themes of epistemic and instrumental rationality are the foundational premise of the site. It is right there in the tagline on the top of the page. I oppose all attempts to replace instrumental rationality with something that involves doing stupid things.
Said literature gives advice, reasoning and conclusions that is epistemically, instrumentally and normatively bad.
This is a recurring issue, so perhaps my instructor and textbooks were atypical: we never discussed or even cared whether someone should defect on PD in my game theory course. The bounds were made clear to us in lecture – game theory studies concepts like Nash equilibria and backward induction (using the term ‘rationality’ to mean VNM-rationality) and applies them to situations like PD; that is all. The use of any normative language in homework sets or exams was pretty much automatically marked incorrect. What one ‘should’ or ‘ought’ to do were instead relegated to other courses in, e.g, economics, philosophy, political science. I’d like to know from others if this is a typical experience from a game theory course (and if anyone happens to be working in the field: if this is representative of the literature).
And the common usage of ‘rational’ on lesswrong should be different to what is used in a significant proportion of game theory literature.
No. The themes of epistemic and instrumental rationality are the foundational premise of the site. It is right there in the tagline on the top of the page. I oppose all attempts to replace instrumental rationality with something that involves doing stupid things.
Upon reflection, I tend to agree with these statements. In this case, perhaps we should taboo ‘rationality’ in its game theoretic meaning – use the phrase ‘VNM-rationality’ whenever that is meant instead of LW’s ‘rationality’.
How about let’s not taboo anything, just make it clear up front what is meant when really necessary. I would prefer that because I think such taboos contribute to the entry barrier for every LW newcomer; I don’t want newcomers used to the game-theoretic jargon to keep unwittingly running afoul of this and getting downvoted.
Perhaps it would be useful if Yvain inserted a clarification of this early in the Sequence.
The use of any normative language in homework sets or exams was pretty much automatically marked incorrect.
The normative claim is one I am making now about the ‘rationality’ theories in question. It is the same kind of normative claim I make when I say “empirical tests are better than beliefs from ad baculum”.
Upon reflection, I tend to agree with these statements. In this case, perhaps we should taboo ‘rationality’ in its game theoretic meaning – use the phrase ‘VNM-rationality’ whenever that is meant instead of LW’s ‘rationality’.
I could agree to that—conditional on confirmation from one of the Vladimirs that the axioms in question do, in fact, imply the faux-rational (CDT like) conclusions the term would be used to represent. I don’t actually see it at a glance and would expect another hidden assumption to be required. I wouldn’t be comfortable using the term without confirmation.
The normative claim is one I am making now about the ‘rationality’ theories in question.
I quoted badly; I believe there was a misunderstanding. The first quote in the parent to this should be taken in the context of your sentence segment that “Said literature gives advice”. In my paragraph, I was objecting to this from my experiences in my course, where I did not receive any advice on what to do in games like PD. Instead, the type of advice that I received was on how to calculate Nash equilibria and find SPNEs.
Otherwise, I am mostly in agreement with the latter part of that sentence. (ETA: That is, I agree that if current game theoretic equilibrium solutions are taken as advice on what one ought to do, then that is often epistemically, instrumentally, and normatively bad.)
More ETA:
conditional on confirmation from one of the Vladimirs that the axioms in question do, in fact, imply the faux-rational (CDT like) conclusions the term would be used to represent. I don’t actually see it at a glance and would expect another hidden assumption to be required.
You are correct – VNM-rationality is incredibly weak (though humans don’t satisfy it). It is, after all, logically equivalent to the existence of a utility function (the proof of this by von Neumann and Morgenstern led to the eponymous VNM theorem). The faux-rationality on LW and in popular culture requires much stronger assumptions. But again, I don’t think these assumptions are made in the game theory literature – I think that faux-rationality is misattributed to game theory. The game theory I was taught used only VNM-rationality, and gave no advice.
Said literature gives advice, reasoning and conclusions that is epistemically, instrumentally and normatively bad.
I remember hearing about studies where economics and game theory students ended up less “moral” by many usual measures after completing their courses. Less inclined to co-operate, more likely to lie and cheat, more concerned about money, more likely to excuse overtly selfish behaviour and so on. And then these fine, new, upstanding citizens, went on to become the next generation of bankers, traders, stock-brokers, and advisers to politicians and industry. The rest as they say is history.
Said literature gives advice, reasoning and conclusions that is epistemically, instrumentally and normatively bad.
Said literature makes statements about what is game-theory-rational. Those statements are only epistemically, instrumentally or normatively bad if you take them to be statements about what is LW-rational or “rational” in the layperson’s sense.
Ideally we’d use different terms for game-theory-rational and LW-rational, but in the meantime we just need to keep the distinction clear in our heads so that we don’t accidentally equivocate between the two.
Said literature makes statements about what is game-theory-rational. Those statements are only epistemically, instrumentally or normatively bad if you take them to be statements about what is LW-rational or “rational” in the layperson’s sense.
Disagree on instrumentally and normatively. Agree regarding epistemically—at least when the works are careful with what claims are made. Also disagree with the “game-theory-rational”, although I understand the principle you are trying to get at. A more limited claim needs to be made or more precise terminology.
I would be interested in reading about the bases for your disagreement. Game theory is essentially the exploration of what happens if you postulate entities who are perfectly informed, personal utility-maximisers who do not care at all either way about other entities. There’s no explicit or implicit claim that people ought to behave like those entities, thus no normative content whatsoever. So I can’t see how the game theory literature could be said to give normatively bad advice, unless the speaker misunderstood the definition of rationality being used, and thought that some definition of rationality was being used in which rationality is normative.
I’m not sure what negative epistemic or instrumental outcomes you foresee either, but I’m open to the possibility that there are some.
Is there a term you prefer to “game-theory-rational” that captures the same meaning? As stated above, game theory is the exploration of what happens when entities that are “rational” by that specific definition interact with the world or each other, so it seems like the ideal term to me.
I would be interested in reading about the bases for your disagreement. Game theory is essentially the exploration of what happens if you postulate entities who are perfectly informed, personal utility-maximisers who do not care at all either way about other entities.
Under this definition you can’t claim epistemic accuracy either. In particular the ‘perfectly informed’ assumption when combined with the personal utility maximization leads to different behaviors to those described as ‘rational’. (It needs to be weakened to “perfectly informed about everything except those parts of the universe that are the other agent.)
There’s no explicit or implicit claim that people ought to behave like those entities, thus no normative content whatsoever.
This isn’t about the agents having selfish desires (in fact, they don’t even have to “not care at all about other entities”—altruism determines what the utility function is, not how to maximise it.) No, this is about shoddy claims about decision theory that are either connotatively misleading or erroneous depending on how they are framed. All those poor paperclip maximisers who read such sources and take them at face value will end up producing less paperclips than they could have if they knew the correct way to interact with the staples maximisers in contrived scenarios.
This isn’t about the agents having selfish desires (in fact, they don’t even have to “not care at all about other entities”—altruism determines what the utility function is, not how to maximise it.)
This is wrong. The standard assumption is that game-theory-rational entities are neither altruistic nor malevolent. Otherwise the Prisoner’s Dilemma wouldn’t be a dilemma in game theory. It’s only a dilemma as long as both players are solely interested in their own outcomes. As soon as you allow players to have altruistic interests in other players’ outcomes it ceases to be a dilemma.
You can do similar mathematical analyses with altruistic agents, but at that point speaking strictly you’re doing decision-theoretic calculations or possibly utilitarian calculations not game-theoretic calculations.
Utilitarian ethics, game theory and decision theory are three different things, and it seems to me your criticism assumes that statements about game theory should be taken as statements about utilitarian ethics or statements about decision theory. I think that is an instance of the fallacy of composition and we’re better served to stay very aware of the distinctions between those three frameworks.
This is wrong. The standard assumption is that game-theory-rational entities are neither altruistic nor malevolent.
No, it just isn’t. Game theory is completely agnostic about what the preferences of the players are based on. Game theory takes a payoff matrix and calculates things like Nash Equilibrium and Dominant Strategies. The verbal description of why the payoff matrix happens to be as it is is fluff.
Otherwise the Prisoner’s Dilemma wouldn’t be a dilemma in game theory. It’s only a dilemma as long as both players are solely interested in their own outcomes. As soon as you allow players to have altruistic interests in other players’ outcomes it ceases to be a dilemma.
As soon as you allow altruistic interests the game ceases to be a Prisoner’s Dilemma. The dilemma (and game theory in general) relies on the players being perfectly selfish in the sense that they ruthlessly maximise their own payoffs as they are defined, not in the sense that those payoffs must never refer to aspects of the universe that happen to include the physical state of the other agents.
Consider the Codependent Prisoner’s Dilemma. Romeo and Juliet have been captured and the guards are trying to extort confessions out of them. However Romeo and Juliet are both lovesick and infatuated and care only about what happens to their lover, not what happens to themselves. Naturally the guards offer Romeo the deal “If you confess we’ll let Juliet go and you’ll get 10 years but if you don’t confess you’ll both get 1 year” (and vice versa, with a both confess clause in there somewhere). Game theory is perfectly equipped at handling this game. In fact, so much so that it wouldn’t even bother calling it a new name. It’s just a Prisoner’s Dilemma and the fact that the conflict of interests between Romeo and Juliet happens to be based on codependent altruism rather than narcissism is outside the scope of what game theorists care about.
It seems from my perspective that we are talking past each other and that your responses are no longer tracking the original point. I don’t personally think that deserves upvotes, but others obviously differ.
Your original claim was that:
Said literature gives advice, reasoning and conclusions that is epistemically, instrumentally and normatively bad.
Now given that game theory is not making any normative claims, it can’t be saying things which are normatively bad. Similarly since game theory does not say that you should either go out and act like a game-theory-rational agent or that you should act as if others will do so, it can’t be saying anything instrumentally bad either.
I just don’t see how it could even be possible for game theory to do what you claim it does. That would be like stating that a document describing the rules of poker was instrumentally and normatively bad because it encouraged wasteful, zero-sum gaming. It would be mistaking description for prescription.
We have already agreed, I think, that there is nothing epistemically bad about game theory taken as it is.
Everything below responds to the off-track discussion above and can be safely ignored by posters not specifically interested in that digression.
In game theory each player’s payoff matrix is their own. Notice that Codependent Romeo does not care where Codependent Juliet ends up in her payoff matrix. If Codependent Romeo was altruistic in the sense of wanting to maximise Juliet’s satisfaction with her payoff, he’d be keeping silent. Because Codependent Romeo is game-theory-rational, he’s indifferent to Codependent Juliet’s satisfaction with her outcome and only cares about maximising his personal payoff.
The standard assumption in a game-theoretic analysis is that a poker player wants money, a chess player wants to win chess games and so on, and that they are indifferent to their opponent(s) opinion about the outcome, just as Codependent Romeo is maximising his own payoff matrix and is indifferent to Codependent Juliet’s.
That is what we attempt to convey when we tell people that game-theory-rational players are neither benevolent nor malevolent. Even if you incorporate something you want to call “altruism” into their preference order, they still don’t care directly about where anyone else ends up in those other peoples’ preference orders.
Now given that game theory is not making any normative claims, it can’t be saying things which are normatively bad.
Not true. The word ‘connotations’ comes to mind. As does “reframing to the extent of outright redefining a critical keyword”. That is not a normatively neutral act. It is legitimate for me to judge it and I choose to do so—negatively.
Agreed.
I want to point out that Eliezer’s (and LW’s general) use of the word ‘rationality’ is entirely different from the use of the word in the game theory literature, where it usually means VNM-rationaliity, or is used to elaborate concepts like sequential rationality in SPNE-type equilibria.
ETA: Reading Grognor’s reply to the parent, it seems that much of the negative affect is due to inconsistent use of the word ‘rational(ity)’ on LW. Maybe it’s time to try yet again to taboo LW’s ‘rationality’ to avoid the namespace collision with academic literature.
And the common usage of ‘rational’ on lesswrong should be different to what is used in a significant proportion of game theory literature. Said literature gives advice, reasoning and conclusions that is epistemically, instrumentally and normatively bad. According to the basic principles of the site it is in fact stupid and not-rational to defect against a clone of yourself in a true Prisoner’s Dilemma. A kind of stupidity that is not too much different to being ‘rational’ like Spock.
No. The themes of epistemic and instrumental rationality are the foundational premise of the site. It is right there in the tagline on the top of the page. I oppose all attempts to replace instrumental rationality with something that involves doing stupid things.
I do endorse avoiding excessive use of the word.
This is a recurring issue, so perhaps my instructor and textbooks were atypical: we never discussed or even cared whether someone should defect on PD in my game theory course. The bounds were made clear to us in lecture – game theory studies concepts like Nash equilibria and backward induction (using the term ‘rationality’ to mean VNM-rationality) and applies them to situations like PD; that is all. The use of any normative language in homework sets or exams was pretty much automatically marked incorrect. What one ‘should’ or ‘ought’ to do were instead relegated to other courses in, e.g, economics, philosophy, political science. I’d like to know from others if this is a typical experience from a game theory course (and if anyone happens to be working in the field: if this is representative of the literature).
Upon reflection, I tend to agree with these statements. In this case, perhaps we should taboo ‘rationality’ in its game theoretic meaning – use the phrase ‘VNM-rationality’ whenever that is meant instead of LW’s ‘rationality’.
How about let’s not taboo anything, just make it clear up front what is meant when really necessary. I would prefer that because I think such taboos contribute to the entry barrier for every LW newcomer; I don’t want newcomers used to the game-theoretic jargon to keep unwittingly running afoul of this and getting downvoted.
Perhaps it would be useful if Yvain inserted a clarification of this early in the Sequence.
The normative claim is one I am making now about the ‘rationality’ theories in question. It is the same kind of normative claim I make when I say “empirical tests are better than beliefs from ad baculum”.
I could agree to that—conditional on confirmation from one of the Vladimirs that the axioms in question do, in fact, imply the faux-rational (CDT like) conclusions the term would be used to represent. I don’t actually see it at a glance and would expect another hidden assumption to be required. I wouldn’t be comfortable using the term without confirmation.
I quoted badly; I believe there was a misunderstanding. The first quote in the parent to this should be taken in the context of your sentence segment that “Said literature gives advice”. In my paragraph, I was objecting to this from my experiences in my course, where I did not receive any advice on what to do in games like PD. Instead, the type of advice that I received was on how to calculate Nash equilibria and find SPNEs.
Otherwise, I am mostly in agreement with the latter part of that sentence. (ETA: That is, I agree that if current game theoretic equilibrium solutions are taken as advice on what one ought to do, then that is often epistemically, instrumentally, and normatively bad.)
More ETA:
You are correct – VNM-rationality is incredibly weak (though humans don’t satisfy it). It is, after all, logically equivalent to the existence of a utility function (the proof of this by von Neumann and Morgenstern led to the eponymous VNM theorem). The faux-rationality on LW and in popular culture requires much stronger assumptions. But again, I don’t think these assumptions are made in the game theory literature – I think that faux-rationality is misattributed to game theory. The game theory I was taught used only VNM-rationality, and gave no advice.
I remember hearing about studies where economics and game theory students ended up less “moral” by many usual measures after completing their courses. Less inclined to co-operate, more likely to lie and cheat, more concerned about money, more likely to excuse overtly selfish behaviour and so on. And then these fine, new, upstanding citizens, went on to become the next generation of bankers, traders, stock-brokers, and advisers to politicians and industry. The rest as they say is history.
Said literature makes statements about what is game-theory-rational. Those statements are only epistemically, instrumentally or normatively bad if you take them to be statements about what is LW-rational or “rational” in the layperson’s sense.
Ideally we’d use different terms for game-theory-rational and LW-rational, but in the meantime we just need to keep the distinction clear in our heads so that we don’t accidentally equivocate between the two.
Disagree on instrumentally and normatively. Agree regarding epistemically—at least when the works are careful with what claims are made. Also disagree with the “game-theory-rational”, although I understand the principle you are trying to get at. A more limited claim needs to be made or more precise terminology.
I would be interested in reading about the bases for your disagreement. Game theory is essentially the exploration of what happens if you postulate entities who are perfectly informed, personal utility-maximisers who do not care at all either way about other entities. There’s no explicit or implicit claim that people ought to behave like those entities, thus no normative content whatsoever. So I can’t see how the game theory literature could be said to give normatively bad advice, unless the speaker misunderstood the definition of rationality being used, and thought that some definition of rationality was being used in which rationality is normative.
I’m not sure what negative epistemic or instrumental outcomes you foresee either, but I’m open to the possibility that there are some.
Is there a term you prefer to “game-theory-rational” that captures the same meaning? As stated above, game theory is the exploration of what happens when entities that are “rational” by that specific definition interact with the world or each other, so it seems like the ideal term to me.
Under this definition you can’t claim epistemic accuracy either. In particular the ‘perfectly informed’ assumption when combined with the personal utility maximization leads to different behaviors to those described as ‘rational’. (It needs to be weakened to “perfectly informed about everything except those parts of the universe that are the other agent.)
This isn’t about the agents having selfish desires (in fact, they don’t even have to “not care at all about other entities”—altruism determines what the utility function is, not how to maximise it.) No, this is about shoddy claims about decision theory that are either connotatively misleading or erroneous depending on how they are framed. All those poor paperclip maximisers who read such sources and take them at face value will end up producing less paperclips than they could have if they knew the correct way to interact with the staples maximisers in contrived scenarios.
This is wrong. The standard assumption is that game-theory-rational entities are neither altruistic nor malevolent. Otherwise the Prisoner’s Dilemma wouldn’t be a dilemma in game theory. It’s only a dilemma as long as both players are solely interested in their own outcomes. As soon as you allow players to have altruistic interests in other players’ outcomes it ceases to be a dilemma.
You can do similar mathematical analyses with altruistic agents, but at that point speaking strictly you’re doing decision-theoretic calculations or possibly utilitarian calculations not game-theoretic calculations.
Utilitarian ethics, game theory and decision theory are three different things, and it seems to me your criticism assumes that statements about game theory should be taken as statements about utilitarian ethics or statements about decision theory. I think that is an instance of the fallacy of composition and we’re better served to stay very aware of the distinctions between those three frameworks.
No, it just isn’t. Game theory is completely agnostic about what the preferences of the players are based on. Game theory takes a payoff matrix and calculates things like Nash Equilibrium and Dominant Strategies. The verbal description of why the payoff matrix happens to be as it is is fluff.
As soon as you allow altruistic interests the game ceases to be a Prisoner’s Dilemma. The dilemma (and game theory in general) relies on the players being perfectly selfish in the sense that they ruthlessly maximise their own payoffs as they are defined, not in the sense that those payoffs must never refer to aspects of the universe that happen to include the physical state of the other agents.
Consider the Codependent Prisoner’s Dilemma. Romeo and Juliet have been captured and the guards are trying to extort confessions out of them. However Romeo and Juliet are both lovesick and infatuated and care only about what happens to their lover, not what happens to themselves. Naturally the guards offer Romeo the deal “If you confess we’ll let Juliet go and you’ll get 10 years but if you don’t confess you’ll both get 1 year” (and vice versa, with a both confess clause in there somewhere). Game theory is perfectly equipped at handling this game. In fact, so much so that it wouldn’t even bother calling it a new name. It’s just a Prisoner’s Dilemma and the fact that the conflict of interests between Romeo and Juliet happens to be based on codependent altruism rather than narcissism is outside the scope of what game theorists care about.
It seems from my perspective that we are talking past each other and that your responses are no longer tracking the original point. I don’t personally think that deserves upvotes, but others obviously differ.
Your original claim was that:
Now given that game theory is not making any normative claims, it can’t be saying things which are normatively bad. Similarly since game theory does not say that you should either go out and act like a game-theory-rational agent or that you should act as if others will do so, it can’t be saying anything instrumentally bad either.
I just don’t see how it could even be possible for game theory to do what you claim it does. That would be like stating that a document describing the rules of poker was instrumentally and normatively bad because it encouraged wasteful, zero-sum gaming. It would be mistaking description for prescription.
We have already agreed, I think, that there is nothing epistemically bad about game theory taken as it is.
Everything below responds to the off-track discussion above and can be safely ignored by posters not specifically interested in that digression.
In game theory each player’s payoff matrix is their own. Notice that Codependent Romeo does not care where Codependent Juliet ends up in her payoff matrix. If Codependent Romeo was altruistic in the sense of wanting to maximise Juliet’s satisfaction with her payoff, he’d be keeping silent. Because Codependent Romeo is game-theory-rational, he’s indifferent to Codependent Juliet’s satisfaction with her outcome and only cares about maximising his personal payoff.
The standard assumption in a game-theoretic analysis is that a poker player wants money, a chess player wants to win chess games and so on, and that they are indifferent to their opponent(s) opinion about the outcome, just as Codependent Romeo is maximising his own payoff matrix and is indifferent to Codependent Juliet’s.
That is what we attempt to convey when we tell people that game-theory-rational players are neither benevolent nor malevolent. Even if you incorporate something you want to call “altruism” into their preference order, they still don’t care directly about where anyone else ends up in those other peoples’ preference orders.
Not true. The word ‘connotations’ comes to mind. As does “reframing to the extent of outright redefining a critical keyword”. That is not a normatively neutral act. It is legitimate for me to judge it and I choose to do so—negatively.