The blackmail letter has someone reading the AI agent’s source code to figure out what it would do, and therefore runs into the objection “you are asserting that the blackmailer can solve the Halting Problem”.
Somewhat. If it is known that the AI actually does not go into infinite loops, then this isn’t a problem—but this creates an interesting question as to how the AI is reasoning about the human’s behavior in a way that doesn’t lead to an infinite loop. One sort of answer we can give is that they’re doing logical reasoning about each other, rather than trying to run each other’s code. This could run into incompleteness problems, but not always:
The blackmail letter has someone reading the AI agent’s source code to figure out what it would do, and therefore runs into the objection “you are asserting that the blackmailer can solve the Halting Problem”.
Somewhat. If it is known that the AI actually does not go into infinite loops, then this isn’t a problem—but this creates an interesting question as to how the AI is reasoning about the human’s behavior in a way that doesn’t lead to an infinite loop. One sort of answer we can give is that they’re doing logical reasoning about each other, rather than trying to run each other’s code. This could run into incompleteness problems, but not always:
http://intelligence.org/files/ParametricBoundedLobsTheorem.pdf