That’s as much as I got. I don’t understand the conditions for a Nash equilibrium to be optimal for everybody, i.e. something the rational agents would actually all want to do instead of something that just has nice properties. You say that “purely rational players will always end up at a Nash equilibrium”, but I don’t think your post quite established that and I don’t think it can be completely true, because of things like the self-PD. Is that what the SI’s open problems of “blackmail-free equilibrium among timeless strategists” and “fair division by continuous / multiparty agents” are about maybe? That would be cool.
“Optimal for everybody” is a very un-game-theoretic outlook. Game theorists think more in terms of “optimal for me, and screw the other guy”. If everyone involved is totally selfish, and they expect other players to be pretty good at figuring out their strategy, and they don’t have some freaky way of correlating their decisions with those of other agents like TDT, then they’ll aim for a Nash equilibrium (though if there are multiple Nash equilibria, they might not hit the same one).
This fails either when agents aren’t totally selfish (if, like you, they’re looking for what’s optimal for everyone, which is a very different problem), or if they’re using an advanced decision theory to correlate their decisions, which is harder for normal people than it is for people playing against clones of themselves or superintelligences that can output their programs. I’ll discuss this more later.
Also, and probably related to not having clear conditions for an optimal equilibrium, the meta-game of submitting strategies doesn’t really look like it solves the infinite regress: if we find ourselves in a Nash equilibrium and agents’ decisions are independent, then great, we don’t have to model anything because the particular blend of utilities and enforced common ignorance means no strategy change is expected to help, but getting everyone into an equilibrium rationally (instead of hoping everyone likes the same flavor of Schelling’s Heuristic Soup) still looks hard.
This fails either when agents aren’t totally selfish (if, like you, they’re looking for what’s optimal for everyone, which is a very different problem)
It’s not very different—you just need to alter the agents’ utility functions slightly, to value the other player gaining utility as well.
E.g. take the standard Prisoner’s Dilemma: 5 for me, 0 for you if I can betray you, 3 each if we cooperate, 1 each if we defect. The equilibrium is defect / defect. Now let’s make our agents’ utility function look altruistic—each agent gets a reward, and then gets to add the opponent’s reward as well (no loops, just add at the end.) Now our payoff is 5+0 for me, 0+5 for you if I betray you, 3+3 each if we cooperate, and 1+1 each if we defect.
A purely self-interested agent with that utility function has the equilibrium at cooperate / cooperate.
More generally, the math in games involved selfish players goes through if you represent their altruism in their own utility function, so they can act still simply pick the highest number ‘selfishly’ .
“Optimal for everybody” is a very un-game-theoretic outlook. Game theorists think more in terms of “optimal for me, and screw the other guy”. If everyone involved is totally selfish, and they expect other players to be pretty good at figuring out their strategy, and they don’t have some freaky way of correlating their decisions with those of other agents like TDT, then they’ll aim for a Nash equilibrium (though if there are multiple Nash equilibria, they might not hit the same one).
This fails either when agents aren’t totally selfish (if, like you, they’re looking for what’s optimal for everyone, which is a very different problem), or if they’re using an advanced decision theory to correlate their decisions, which is harder for normal people than it is for people playing against clones of themselves or superintelligences that can output their programs. I’ll discuss this more later.
Agreed.
It’s not very different—you just need to alter the agents’ utility functions slightly, to value the other player gaining utility as well.
E.g. take the standard Prisoner’s Dilemma: 5 for me, 0 for you if I can betray you, 3 each if we cooperate, 1 each if we defect. The equilibrium is defect / defect. Now let’s make our agents’ utility function look altruistic—each agent gets a reward, and then gets to add the opponent’s reward as well (no loops, just add at the end.) Now our payoff is 5+0 for me, 0+5 for you if I betray you, 3+3 each if we cooperate, and 1+1 each if we defect.
A purely self-interested agent with that utility function has the equilibrium at cooperate / cooperate.
More generally, the math in games involved selfish players goes through if you represent their altruism in their own utility function, so they can act still simply pick the highest number ‘selfishly’ .
-