I don’t think anybody is suggesting building an explicit “just say ‘No’ to extortion” heuristic into an AI. (I agree we do not have a good definition of “extortion” so when I use the word I use it in an intuitive sense.) We’re trying to find a general decision theory that naturally ends up saying no to extortion (when it makes sense to).
That is pretty incoherent. If you are trying to come up with a general decision theory that wins and also says no to extortion, then you have overdetermined the problem (or will overdetermine it once you supply your definition). If you are predicting that a decision theory that wins will say no to extortion, then it is a rather pointless claim until you supply a definition. Perhaps what you really intend to do is to define ‘extortion’ as ‘that which a winning decision theory says no to’. In which case, Nash has defined ‘extortion’ for you—as a threat which is not credible, in his technical sense.
ETA: Of course the [informal] concept of “credibility” breaks down a bit when all agents are reasoning this way. Which is why the problem is still unsolved!
Why do you say the problem is still unsolved? What issues do you feel were not addressed by Nash in 1953? Where is the flaw in his argument?
Part of the difficulty of discussing this here is that you have now started to use the word “credible” informally, when it also has a technical meaning in this context.
That is pretty incoherent. If you are trying to come up with a general decision theory that wins and also says no to extortion, then you have overdetermined the problem (or will overdetermine it once you supply your definition). If you are predicting that a decision theory that wins will say no to extortion, then it is a rather pointless claim until you supply a definition. Perhaps what you really intend to do is to define ‘extortion’ as ‘that which a winning decision theory says no to’. In which case, Nash has defined ‘extortion’ for you—as a threat which is not credible, in his technical sense.
Why do you say the problem is still unsolved? What issues do you feel were not addressed by Nash in 1953? Where is the flaw in his argument?
Part of the difficulty of discussing this here is that you have now started to use the word “credible” informally, when it also has a technical meaning in this context.