Alice threatens Bob when Alice says that, if Bob performs some action X, then Alice will respond with action Y, where Y (a) harms Bob and (b) harms Alice. (If one wants to be “mathematical”, then one could say that each combination of actions is associated with a set of payoffs, and that “action Y harms Bob” == “[Bob’s payoff with Y] < [Bob’s payoff with not-Y]”.)
Note that the dath ilan “negotiation algorithm” arguably fits this definition of “threat”:
If Alis and Bohob both do an equal amount of labor to gain a previously unclaimed resource worth 10 value-units, and Alis has to propose a division of the resource, and Bohob can either accept that division or say they both get nothing, and Alis proposes that Alis get 6 units and Bohob get 4 units, Bohob should accept this proposal with probability < 5⁄6 so Alis’s expected gain from this unfair policy is less than her gain from proposing the fair division of 5 units apiece.
Because for X=”proposes that Alis get 6 units and Bohob get 4 units” and Y=”accepting the proposal with probability < 5/6″, if Alis performs X, then Y harms both Alis and Bohob relative to not-Y (accepting the proposal with probability 1).
So I’m guessing that Eliezer is using some definition of “threat” that refers to “fairness”, such that “fair” actions do not count as threats according to his definition.
By this definition any statement that sets any conditions whatsoever in the Ultimatum Game is a threat. Or indeed any statement setting conditions under which you might withdraw from otherwise mutually beneficial trade.
I think, if there is any way to interpret any such statements as not being a threat, it would be of the form “I have already made my precommitments; I’ve already altered my brain so that I assign lower payoffs (due to psychological pain or whatever) to the outcomes where I fail to carry out my threat. I’m not making a new strategic move; I’m informing you of a past strategic move.” One could argue that the game is no longer the Ultimatum Game, due to the payoffs not being what they are in the Ultimatum Game.
Of course, both sides would like to do this, and to be “first” to do it. An extreme person in this vein could say “I’ve altered my brain so that I will reject anything less than 9-1 in my favor”, and this could even be true. Two such people would be guaranteed to have a bad time if they ran into one another, and a fairly bad time if they met a dath ilani; but one could choose to be such a person.
If both sides do set up their psychology well in advance of encountering the game, then the strategic moves are effectively made simultaneously. One can then think about the game of “making your strategic move”.
Eliezer is using some definition of “threat” that refers to “fairness”, such that “fair” actions do not count as threats
This seems likely. Much of Eliezer’s fiction includes a lot of typical mind fallacy and a seemingly-willful ignorance of power dynamics and “unfair” results in equilibria being the obvious outcome for unaligned agents with different starting conditions.
This kind of game-theory analysis is just silly unless it includes the information about who has the stronger/more-visible precommittments, and what extra-game impacts the actions will have. It’s actually quite surprising how deeply CDT is assumed (agents can freely choose their actions at the point in the narrative where it happens) in such analyses.
It’s hardly a wilful ignorance, it’s a deliberate rejection. A good decision theory, by nature, should produce results that don’t actually depend on visible precommitments to achieve negotiation equilibrium, since an ideal agent negotiating ought to be able to accept postcommitment to things you would predictably wish you’d precommitted to. And if a decision theory doesn’t allow you to hold out for fairness in the face of an uneven power dynamic, why even have one?
Note that the dath ilan “negotiation algorithm” arguably fits this definition of “threat”:
Because for X=”proposes that Alis get 6 units and Bohob get 4 units” and Y=”accepting the proposal with probability < 5/6″, if Alis performs X, then Y harms both Alis and Bohob relative to not-Y (accepting the proposal with probability 1).
So I’m guessing that Eliezer is using some definition of “threat” that refers to “fairness”, such that “fair” actions do not count as threats according to his definition.
By this definition any statement that sets any conditions whatsoever in the Ultimatum Game is a threat. Or indeed any statement setting conditions under which you might withdraw from otherwise mutually beneficial trade.
This is true.
I think, if there is any way to interpret any such statements as not being a threat, it would be of the form “I have already made my precommitments; I’ve already altered my brain so that I assign lower payoffs (due to psychological pain or whatever) to the outcomes where I fail to carry out my threat. I’m not making a new strategic move; I’m informing you of a past strategic move.” One could argue that the game is no longer the Ultimatum Game, due to the payoffs not being what they are in the Ultimatum Game.
Of course, both sides would like to do this, and to be “first” to do it. An extreme person in this vein could say “I’ve altered my brain so that I will reject anything less than 9-1 in my favor”, and this could even be true. Two such people would be guaranteed to have a bad time if they ran into one another, and a fairly bad time if they met a dath ilani; but one could choose to be such a person.
If both sides do set up their psychology well in advance of encountering the game, then the strategic moves are effectively made simultaneously. One can then think about the game of “making your strategic move”.
This seems likely. Much of Eliezer’s fiction includes a lot of typical mind fallacy and a seemingly-willful ignorance of power dynamics and “unfair” results in equilibria being the obvious outcome for unaligned agents with different starting conditions.
This kind of game-theory analysis is just silly unless it includes the information about who has the stronger/more-visible precommittments, and what extra-game impacts the actions will have. It’s actually quite surprising how deeply CDT is assumed (agents can freely choose their actions at the point in the narrative where it happens) in such analyses.
It’s hardly a wilful ignorance, it’s a deliberate rejection. A good decision theory, by nature, should produce results that don’t actually depend on visible precommitments to achieve negotiation equilibrium, since an ideal agent negotiating ought to be able to accept postcommitment to things you would predictably wish you’d precommitted to. And if a decision theory doesn’t allow you to hold out for fairness in the face of an uneven power dynamic, why even have one?