Okay, I’ll be more explicit: I am considering the class of agents who behave one way if they predict you’re muggable and behave another way if they predict you’re unmuggable. The dual of an agent behaves exactly the same as the original agent, except the behaviors are reversed. In symbols:
An agent A has two behaviors.
It it predicts you’d give Omega $5, it will exhibit behavior X; otherwise, it will exhibit behavior Y.
The dual agent A* exhibits behavior Y if it predicts you’d give Omega $5, and X otherwise.
A and A* are equally likely in my prior.
What about Omega?
Omega has two behaviors.
If it predicts you’d give Omega $5, it will flip a coin and give you $100 on heads; otherwise, nothing. In either case, it will tell you the rules of the game.
What would Omega* be?
If Omega predicts you’d give Omega $5, it will do nothing. Otherwise, it will flip a coin and give you $100 on heads. In either case, it will assure you that it is Omega, not Omega.
So the dual of Omega is something that looks like Omega but is in fact deceptive. By hypothesis, Omega is trustworthy, so my prior probability of encountering Omega* is negligible compared to meeting Omega.
(So yeah, there is a dual of Omega, but it’s much less probable than Omega.)
Then, when I calculate expected utility, each agent A is balanced by its dual A , but Omega is not balanced by Omega.
If we assume you can tell “deceptive” agents from “non-deceptive” ones and shift probability weight accordingly, then not every agent is balanced by its dual, because some “deceptive” agents probably have “non-deceptive” duals and vice versa. No?
(Apologies if I’m misunderstanding—this stuff is slowly getting too complex for me to grasp.)
The reason we shift probability weight away from the deceptive Omega is that, in the original problem, we are told that we believe Omega to be non-deceptive. The reasoning goes like this: If it looks like Omega and talks like Omega, then it might be Omega or Omega . But if it were Omega* , then it would be deceiving us, so it’s most probably Omega.
In the original problem, we have no reason to believe that No-mega and friends are non-deceptive.
(But if we did, then yes, the dual of a non-deceptive agent would be deceptive, and so have lower prior probability. This would be a different problem, but it would still have a symmetry: We would have to define a different notion of dual, where the dual of an agent has the reversed behavior and also reverses its claims about its own behavior.
What would Omega* be in that case? It would not claim to be Omega. It would truthfully tell you that if it predicted you would not give it $5 on tails, then it would flip a coin and give you $100 on heads; and otherwise it would not give you anything. This has no bearing on your decision in the Omega problem.)
By your definitions, Omega would condition its decision on you being counterfactually muggable by the original Omega, not on you giving money to Omega itself. Or am I losing the plot again? This notion of “duality” seems to be getting more and more complex.
“Duality” has become more complex because we’re now talking about a more complex problem — a version of Counterfactual Mugging where you believe that all superintelligent agents are trustworthy. The old version of duality suffices for the ordinary Counterfactual Mugging problem.
My thesis is that there’s always a symmetry in the space of black swans like No-mega.
In the case currently under consideration, I’m assuming Omega’s spiel goes something like “I just flipped a coin. If it had been heads, I would have predicted what you would do if I had approached you and given my spiel....” Notice the use of first-person pronouns. Omega* would have almost the same spiel verbatim, also using first-person pronouns, and make no reference to Omega. And, being non-deceptive, it would behave the way it says it does. So it wouldn’t condition on your being muggable by Omega.
You could object to this by claiming that Omega actually says “I am Omega. If Omega had come up to you and said....”, in which case I can come up with a third notion of duality.
If Omega* makes no reference to the original Omega, I don’t understand why they have “opposite behavior with respect to my status as being counterfactually-muggable” (by the original Omega), which was your reason for inventing “duality” in the first place. I apologize, but at this point it’s unclear to me that you actually have a proof of anything. Maybe we can take this discussion to email?
Why? Can’t your definition of dual be applied to Omega? I admit I don’t completely understand the argument.
Okay, I’ll be more explicit: I am considering the class of agents who behave one way if they predict you’re muggable and behave another way if they predict you’re unmuggable. The dual of an agent behaves exactly the same as the original agent, except the behaviors are reversed. In symbols:
An agent A has two behaviors.
It it predicts you’d give Omega $5, it will exhibit behavior X; otherwise, it will exhibit behavior Y.
The dual agent A* exhibits behavior Y if it predicts you’d give Omega $5, and X otherwise.
A and A* are equally likely in my prior.
What about Omega?
Omega has two behaviors.
If it predicts you’d give Omega $5, it will flip a coin and give you $100 on heads; otherwise, nothing. In either case, it will tell you the rules of the game.
What would Omega* be?
If Omega predicts you’d give Omega $5, it will do nothing. Otherwise, it will flip a coin and give you $100 on heads. In either case, it will assure you that it is Omega, not Omega.
So the dual of Omega is something that looks like Omega but is in fact deceptive. By hypothesis, Omega is trustworthy, so my prior probability of encountering Omega* is negligible compared to meeting Omega.
(So yeah, there is a dual of Omega, but it’s much less probable than Omega.)
Then, when I calculate expected utility, each agent A is balanced by its dual A , but Omega is not balanced by Omega.
If we assume you can tell “deceptive” agents from “non-deceptive” ones and shift probability weight accordingly, then not every agent is balanced by its dual, because some “deceptive” agents probably have “non-deceptive” duals and vice versa. No?
(Apologies if I’m misunderstanding—this stuff is slowly getting too complex for me to grasp.)
The reason we shift probability weight away from the deceptive Omega is that, in the original problem, we are told that we believe Omega to be non-deceptive. The reasoning goes like this: If it looks like Omega and talks like Omega, then it might be Omega or Omega . But if it were Omega* , then it would be deceiving us, so it’s most probably Omega.
In the original problem, we have no reason to believe that No-mega and friends are non-deceptive.
(But if we did, then yes, the dual of a non-deceptive agent would be deceptive, and so have lower prior probability. This would be a different problem, but it would still have a symmetry: We would have to define a different notion of dual, where the dual of an agent has the reversed behavior and also reverses its claims about its own behavior.
What would Omega* be in that case? It would not claim to be Omega. It would truthfully tell you that if it predicted you would not give it $5 on tails, then it would flip a coin and give you $100 on heads; and otherwise it would not give you anything. This has no bearing on your decision in the Omega problem.)
Edit: Formatting.
By your definitions, Omega would condition its decision on you being counterfactually muggable by the original Omega, not on you giving money to Omega itself. Or am I losing the plot again? This notion of “duality” seems to be getting more and more complex.
“Duality” has become more complex because we’re now talking about a more complex problem — a version of Counterfactual Mugging where you believe that all superintelligent agents are trustworthy. The old version of duality suffices for the ordinary Counterfactual Mugging problem.
My thesis is that there’s always a symmetry in the space of black swans like No-mega.
In the case currently under consideration, I’m assuming Omega’s spiel goes something like “I just flipped a coin. If it had been heads, I would have predicted what you would do if I had approached you and given my spiel....” Notice the use of first-person pronouns. Omega* would have almost the same spiel verbatim, also using first-person pronouns, and make no reference to Omega. And, being non-deceptive, it would behave the way it says it does. So it wouldn’t condition on your being muggable by Omega.
You could object to this by claiming that Omega actually says “I am Omega. If Omega had come up to you and said....”, in which case I can come up with a third notion of duality.
If Omega* makes no reference to the original Omega, I don’t understand why they have “opposite behavior with respect to my status as being counterfactually-muggable” (by the original Omega), which was your reason for inventing “duality” in the first place. I apologize, but at this point it’s unclear to me that you actually have a proof of anything. Maybe we can take this discussion to email?