If you create your AI before you can infer from Omega’s actions what the umpteenth digit of pi is, then I agree that you should create an AI that presses the button, even if the AI finds out (through Omega’s actions) that the digit is in fact odd.
If you can somehow figure it out, then yes, you shouldn’t press the button. If you know that the simulated you would’ve known to press the button when you don’t, you’re not anymore dealing with “take either 50% chance of world exploding right now VS. 50% chance of world exploding million years from now”, but a lot simpler “I offer to destroy the world, you can say yes or no”. Updateless agent would naturally want to take the winning bet if gaining that information were somehow possible.
So, if you know which digit omega used to decide his actions, and how, and you happen to know that digit, the bet you’re taking is the simpler one, the one where you can simply answer ‘yes’ or ‘no’. Observing that Earth has not been destroyed is not enough evidence though, because the simulated, non-conscious you would’ve observed roughly the same thing. Only if there were some sort of difference that you knew you could and would use in the simulation, like, your knowledge of umpteenth digit of pi, or color of some object in the sky(we’re assuming Omega tells you this much in both cases. This about the fate of humanity, you should seriously be certain about what sort of bets you’re taking.
If you can somehow figure it out, then yes, you shouldn’t press the button. If you know that the simulated you would’ve known to press the button when you don’t, you’re not anymore dealing with “take either 50% chance of world exploding right now VS. 50% chance of world exploding million years from now”, but a lot simpler “I offer to destroy the world, you can say yes or no”. Updateless agent would naturally want to take the winning bet if gaining that information were somehow possible.
So, if you know which digit omega used to decide his actions, and how, and you happen to know that digit, the bet you’re taking is the simpler one, the one where you can simply answer ‘yes’ or ‘no’. Observing that Earth has not been destroyed is not enough evidence though, because the simulated, non-conscious you would’ve observed roughly the same thing. Only if there were some sort of difference that you knew you could and would use in the simulation, like, your knowledge of umpteenth digit of pi, or color of some object in the sky(we’re assuming Omega tells you this much in both cases. This about the fate of humanity, you should seriously be certain about what sort of bets you’re taking.