The reason this seems wrong is because you’re leaving out a crucial step: generating a probability distribution over the set of worlds you could be in, predicated on what you’ve seen already. When you meet a supposed Omega, you should start off believing that he’s a human trying to fool you, which means that you should not pay up in counterfactual mugging and should two-box in Newcomb’s problem. It takes an extraordinary amount of evidence to dislodge that conclusion even a little. Only after you have that evidence can you accurately map out the remaining possibilities. Either Omega gives you the outcome based on your decision as he claims, he gives you a fixed outcome regardless of your decision, or he deviates the algorithm he claims to follow in an obvious way (he doesn’t pay up when he claims he would), or he deviates from the algorithm he claims to follow in a non-obvious way (such as an easter-egg bonus or penalty given when your behavior matches a particular pattern). Once you have probabilities for each of those, you just plug the scenarios into your utility function, multiply by the probability, sum up the expected utility for each decision algorithm, and follow whichever gives the highest expected utility.
The reason this seems wrong is because you’re leaving out a crucial step: generating a probability distribution over the set of worlds you could be in, predicated on what you’ve seen already. When you meet a supposed Omega, you should start off believing that he’s a human trying to fool you, which means that you should not pay up in counterfactual mugging and should two-box in Newcomb’s problem. It takes an extraordinary amount of evidence to dislodge that conclusion even a little. Only after you have that evidence can you accurately map out the remaining possibilities. Either Omega gives you the outcome based on your decision as he claims, he gives you a fixed outcome regardless of your decision, or he deviates the algorithm he claims to follow in an obvious way (he doesn’t pay up when he claims he would), or he deviates from the algorithm he claims to follow in a non-obvious way (such as an easter-egg bonus or penalty given when your behavior matches a particular pattern). Once you have probabilities for each of those, you just plug the scenarios into your utility function, multiply by the probability, sum up the expected utility for each decision algorithm, and follow whichever gives the highest expected utility.