What if we ignore the VR question? Omega tells you that killing and eating your children will make you maximally happy. Should you do it?
Omega can’t tell you that doing X makes you maximally happy unless doing X actually makes you maximally happy. And a scenario where doing X actually makes you maximally happy may be a scenario where you are no longer human and don’t have human preferences.
Omega could, of course, also say “you are mistaken when you conclude that being maximally happy in this scenario is not a human preference”. However,
This conclusion that that is not a human preference is being made by you, the reader, not just by the person in the scenario. It is not possible to stipulate that you, the reader, are wrong about your analysis of some scenario.
Even within the scenario, if someone is mistaken about something like this, it’s a scenario where he can’t trust his own reasoning abilities, so there’s really nothing he can conclude about anything at all. (What if Omega tells you that you don’t understand logic and that every use of logic you think you have done was either wrong or true only by coincidence?)
What if we ignore the VR question? Omega tells you that killing and eating your children will make you maximally happy. Should you do it?
Omega can’t tell you that doing X makes you maximally happy unless doing X actually makes you maximally happy. And a scenario where doing X actually makes you maximally happy may be a scenario where you are no longer human and don’t have human preferences.
Omega could, of course, also say “you are mistaken when you conclude that being maximally happy in this scenario is not a human preference”. However,
This conclusion that that is not a human preference is being made by you, the reader, not just by the person in the scenario. It is not possible to stipulate that you, the reader, are wrong about your analysis of some scenario.
Even within the scenario, if someone is mistaken about something like this, it’s a scenario where he can’t trust his own reasoning abilities, so there’s really nothing he can conclude about anything at all. (What if Omega tells you that you don’t understand logic and that every use of logic you think you have done was either wrong or true only by coincidence?)