So if I understand this correctly, Alice and the Sovereign are identically omniscient, and the Sovereign additionally has some power and influence upon the world that Alice does not. In the case where Alice herself is the sovereign the problem is solved, right? The sovereign just has to figure out what she prefers and do that. The solution then, is to simulate the scenario where Alice has the power to make the decision herself and then match Alice’s decision. This solves both 1 and 2.
My short answer to the broader “How do we know what sacks of meat / circuits / whatever prefer” question is “you look at the behavioral output”. Here, if Alice can make the decision herself, the decision represents her behavioral output.
(I’m about halfway through writing about how to make this idea more workable without resorting to omniscient things with consistent preferences, if I still like the idea after writing it out I’ll cross post it on lw.)
So if I understand this correctly, Alice and the Sovereign are identically omniscient, and the Sovereign additionally has some power and influence upon the world that Alice does not. In the case where Alice herself is the sovereign the problem is solved, right? The sovereign just has to figure out what she prefers and do that. The solution then, is to simulate the scenario where Alice has the power to make the decision herself and then match Alice’s decision. This solves both 1 and 2.
My short answer to the broader “How do we know what sacks of meat / circuits / whatever prefer” question is “you look at the behavioral output”. Here, if Alice can make the decision herself, the decision represents her behavioral output.
(I’m about halfway through writing about how to make this idea more workable without resorting to omniscient things with consistent preferences, if I still like the idea after writing it out I’ll cross post it on lw.)