For any decision theory, isn’t there some hypothetical where Omega can say, “I’ve analyzed your decision theory, and I’m giving you proposition X, such that if you act the way your decision theory believes is optimal, you will lose?” The “Omega scans your brain and tortures you if you’re too rational” would be an obvious example of this.
This isn’t obvious. In particular, note that your “obvious example” violates the basic assumption all these attempts at a decision theory are using, that the payoff depends only on your choice and not how you arrived at it. Of course this is not necessarily a realistic assumption, but that is, IINM, the problem they’re trying to solve.
This isn’t obvious. In particular, note that your “obvious example” violates the basic assumption all these attempts at a decision theory are using, that the payoff depends only on your choice and not how you arrived at it.
Omega simulates you in a variety of scenarios. If you consistently make rational decisions he tortures you.
That does make it somewhat more useful, if that’s the constraint under which it’s operating. It still strikes me as probable that, insofar as decision theory A+ makes decisions that theory A- does not, there must be some way to reward A- and punish A+. I may well be wrong about this. The other flaws, namely the fact that actual decision makers do not encounter omniscient entities with entirely inscrutable motives that are unwaveringly honest, still seem to render the pursuit futile. It’s decidedly less futile if Omega is constrained to outcome based reward/punishment.
This isn’t obvious. In particular, note that your “obvious example” violates the basic assumption all these attempts at a decision theory are using, that the payoff depends only on your choice and not how you arrived at it. Of course this is not necessarily a realistic assumption, but that is, IINM, the problem they’re trying to solve.
Omega simulates you in a variety of scenarios. If you consistently make rational decisions he tortures you.
My reply to this was going to be essentially the same as my comment on bentarm’s thread, so I’ll just point you there.
That does make it somewhat more useful, if that’s the constraint under which it’s operating. It still strikes me as probable that, insofar as decision theory A+ makes decisions that theory A- does not, there must be some way to reward A- and punish A+. I may well be wrong about this. The other flaws, namely the fact that actual decision makers do not encounter omniscient entities with entirely inscrutable motives that are unwaveringly honest, still seem to render the pursuit futile. It’s decidedly less futile if Omega is constrained to outcome based reward/punishment.