Nice post, thanks for writing it up. It was interesting and very easy to read because principles were all first motivated by problems.
That said, I still feel confused. Let me try to re-frame your post in a decision theory setting: We have to make a decision, so the infinite regress of dependencies from rational players modelling each other, which doesn’t generally settle to a fixed set of actions for all players who model each other naively (as shown by the penny matching example), has to be truncated some how. To do that, the game theory people decided that they’d build a new game where instead of rational players picking actions, rational players choose among strategies which return irrational players who take constant actions. These strategies make use of “biased dice”, which are facts whose values the rational players are ignorant of, at least until they’ve selected their strategies. Each player gets their own dice of whatever biases they want, and all the dice are independent of each other. By introducing the dice, a rational player makes themself and all the other players partially uncertain about what constant action player they will select. This common enforced ignorance (or potential ignorance, since the rational players can submit pure strategies and skip the dice all together) can have some interesting properties: there will always be (for games with whatever reasonable conditions) at least one so called Nash equilibrium, which is a set of strategies returned by the rational players for which no player would unilaterally modify their decision upon learning the decisions of the others (though this and the Schelling point thing sound like an inconsistent bag of features from people who aren’t sure whether the agents’ decisions do or don’t depend on each other).
That’s as much as I got. I don’t understand the conditions for a Nash equilibrium to be optimal for everybody, i.e. something the rational agents would actually all want to do instead of something that just has nice properties. You say that “purely rational players will always end up at a Nash equilibrium”, but I don’t think your post quite established that and I don’t think it can be completely true, because of things like the self-PD. Is that what the SI’s open problems of “blackmail-free equilibrium among timeless strategists” and “fair division by continuous / multiparty agents” are about maybe? That would be cool.
Also, and probably related to not having clear conditions for an optimal equilibrium, the meta-game of submitting strategies doesn’t really look like it solves the infinite regress: if we find ourselves in a Nash equilibrium and agents’ decisions are independent, then great, we don’t have to model anything because the particular blend of utilities and enforced common ignorance means no strategy change is expected to help, but getting everyone into an equilibrium rationally (instead of hoping everyone likes the same flavor of Schelling’s Heuristic Soup) still looks hard.
If Nash equilibrium is about indifference of unilateral strategy changes for independent decisions, are there ever situations where a player can switch to a strategy with equal expected utility, putting the strategy set in disequilibrium, so that other players could change their strategies, with all players improving or keeping their expected utility constant, putting the strategy set in a different equilibrium (or into a strategy set where everyone is better off, like mutual cooperation in PD)? That is, are there negotiation paths out of equilibria whose steps make no one worse off? If so, it’d then be nice to find a general selfish scheme by which players will limit themselves to those paths. This sounds like a big fun problem.
That’s as much as I got. I don’t understand the conditions for a Nash equilibrium to be optimal for everybody, i.e. something the rational agents would actually all want to do instead of something that just has nice properties. You say that “purely rational players will always end up at a Nash equilibrium”, but I don’t think your post quite established that and I don’t think it can be completely true, because of things like the self-PD. Is that what the SI’s open problems of “blackmail-free equilibrium among timeless strategists” and “fair division by continuous / multiparty agents” are about maybe? That would be cool.
“Optimal for everybody” is a very un-game-theoretic outlook. Game theorists think more in terms of “optimal for me, and screw the other guy”. If everyone involved is totally selfish, and they expect other players to be pretty good at figuring out their strategy, and they don’t have some freaky way of correlating their decisions with those of other agents like TDT, then they’ll aim for a Nash equilibrium (though if there are multiple Nash equilibria, they might not hit the same one).
This fails either when agents aren’t totally selfish (if, like you, they’re looking for what’s optimal for everyone, which is a very different problem), or if they’re using an advanced decision theory to correlate their decisions, which is harder for normal people than it is for people playing against clones of themselves or superintelligences that can output their programs. I’ll discuss this more later.
Also, and probably related to not having clear conditions for an optimal equilibrium, the meta-game of submitting strategies doesn’t really look like it solves the infinite regress: if we find ourselves in a Nash equilibrium and agents’ decisions are independent, then great, we don’t have to model anything because the particular blend of utilities and enforced common ignorance means no strategy change is expected to help, but getting everyone into an equilibrium rationally (instead of hoping everyone likes the same flavor of Schelling’s Heuristic Soup) still looks hard.
This fails either when agents aren’t totally selfish (if, like you, they’re looking for what’s optimal for everyone, which is a very different problem)
It’s not very different—you just need to alter the agents’ utility functions slightly, to value the other player gaining utility as well.
E.g. take the standard Prisoner’s Dilemma: 5 for me, 0 for you if I can betray you, 3 each if we cooperate, 1 each if we defect. The equilibrium is defect / defect. Now let’s make our agents’ utility function look altruistic—each agent gets a reward, and then gets to add the opponent’s reward as well (no loops, just add at the end.) Now our payoff is 5+0 for me, 0+5 for you if I betray you, 3+3 each if we cooperate, and 1+1 each if we defect.
A purely self-interested agent with that utility function has the equilibrium at cooperate / cooperate.
More generally, the math in games involved selfish players goes through if you represent their altruism in their own utility function, so they can act still simply pick the highest number ‘selfishly’ .
Nice post, thanks for writing it up. It was interesting and very easy to read because principles were all first motivated by problems.
That said, I still feel confused. Let me try to re-frame your post in a decision theory setting: We have to make a decision, so the infinite regress of dependencies from rational players modelling each other, which doesn’t generally settle to a fixed set of actions for all players who model each other naively (as shown by the penny matching example), has to be truncated some how. To do that, the game theory people decided that they’d build a new game where instead of rational players picking actions, rational players choose among strategies which return irrational players who take constant actions. These strategies make use of “biased dice”, which are facts whose values the rational players are ignorant of, at least until they’ve selected their strategies. Each player gets their own dice of whatever biases they want, and all the dice are independent of each other. By introducing the dice, a rational player makes themself and all the other players partially uncertain about what constant action player they will select. This common enforced ignorance (or potential ignorance, since the rational players can submit pure strategies and skip the dice all together) can have some interesting properties: there will always be (for games with whatever reasonable conditions) at least one so called Nash equilibrium, which is a set of strategies returned by the rational players for which no player would unilaterally modify their decision upon learning the decisions of the others (though this and the Schelling point thing sound like an inconsistent bag of features from people who aren’t sure whether the agents’ decisions do or don’t depend on each other).
That’s as much as I got. I don’t understand the conditions for a Nash equilibrium to be optimal for everybody, i.e. something the rational agents would actually all want to do instead of something that just has nice properties. You say that “purely rational players will always end up at a Nash equilibrium”, but I don’t think your post quite established that and I don’t think it can be completely true, because of things like the self-PD. Is that what the SI’s open problems of “blackmail-free equilibrium among timeless strategists” and “fair division by continuous / multiparty agents” are about maybe? That would be cool.
Also, and probably related to not having clear conditions for an optimal equilibrium, the meta-game of submitting strategies doesn’t really look like it solves the infinite regress: if we find ourselves in a Nash equilibrium and agents’ decisions are independent, then great, we don’t have to model anything because the particular blend of utilities and enforced common ignorance means no strategy change is expected to help, but getting everyone into an equilibrium rationally (instead of hoping everyone likes the same flavor of Schelling’s Heuristic Soup) still looks hard.
If Nash equilibrium is about indifference of unilateral strategy changes for independent decisions, are there ever situations where a player can switch to a strategy with equal expected utility, putting the strategy set in disequilibrium, so that other players could change their strategies, with all players improving or keeping their expected utility constant, putting the strategy set in a different equilibrium (or into a strategy set where everyone is better off, like mutual cooperation in PD)? That is, are there negotiation paths out of equilibria whose steps make no one worse off? If so, it’d then be nice to find a general selfish scheme by which players will limit themselves to those paths. This sounds like a big fun problem.
“Optimal for everybody” is a very un-game-theoretic outlook. Game theorists think more in terms of “optimal for me, and screw the other guy”. If everyone involved is totally selfish, and they expect other players to be pretty good at figuring out their strategy, and they don’t have some freaky way of correlating their decisions with those of other agents like TDT, then they’ll aim for a Nash equilibrium (though if there are multiple Nash equilibria, they might not hit the same one).
This fails either when agents aren’t totally selfish (if, like you, they’re looking for what’s optimal for everyone, which is a very different problem), or if they’re using an advanced decision theory to correlate their decisions, which is harder for normal people than it is for people playing against clones of themselves or superintelligences that can output their programs. I’ll discuss this more later.
Agreed.
It’s not very different—you just need to alter the agents’ utility functions slightly, to value the other player gaining utility as well.
E.g. take the standard Prisoner’s Dilemma: 5 for me, 0 for you if I can betray you, 3 each if we cooperate, 1 each if we defect. The equilibrium is defect / defect. Now let’s make our agents’ utility function look altruistic—each agent gets a reward, and then gets to add the opponent’s reward as well (no loops, just add at the end.) Now our payoff is 5+0 for me, 0+5 for you if I betray you, 3+3 each if we cooperate, and 1+1 each if we defect.
A purely self-interested agent with that utility function has the equilibrium at cooperate / cooperate.
More generally, the math in games involved selfish players goes through if you represent their altruism in their own utility function, so they can act still simply pick the highest number ‘selfishly’ .
-