If one defines rationality in some way that isn’t about winning, your example shows that rationalists-in-such-a-sense might not win.
If one defines rationality as actually winning, your example shows that there are things that even Omega cannot do because they involve logical contradiction.
If one defines rationality as something like “expected winning given one’s model of the universe” (for quibbles, see below), your example shows that you can’t coherently carry around a model of the universe that includes a superbeing who deliberately acts so as to invalidate that model.
I find all three of these things rather unsurprising.
The traditional form of Newcomb’s problem doesn’t involve a superbeing deliberately acting so as to invalidate your model of the universe. That seems like a big enough difference from your version to invalidate inferences of the form “there’s no such thing as acting rationally in grobstein’s version of Newcomb’s problem; therefore it doesn’t make sense to use any version of Newcomb’s problem in forming one’s ideas about what constitutes acting rationally”.
I think the third definition is pretty much what Eliezer is getting at when he declares that rationalists/rationality should win. Tightening it up a bit, I think we get something like this: rationality is a strategy S such that at each moment, acting as S tells you to act—given (1) your beliefs about the universe at that point and (2) your intention of following S at all times—maximizes your net utility (calculated in whatever way you prefer; that is mostly not a question of rationality). This isn’t quite a definition, because there might turn out to be multiple such strategies, especially for people whose initial beliefs about the universe are sufficiently crazy. But if you add some condition to the effect that S and your initial beliefs shouldn’t be too unlike what’s generally considered (respectively) rational and right now, there might well be a unique solution to the equations. And it seems to me that what your example shows about this definition is simply that you can’t consistently expect Omega to act in a way that falsifies your beliefs and/or invalidates your strategies for acting. Which is (to me) not surprising, and not a reason for defining rationality in this sort of way.
Can he not scan your brain and determine what strategy you are following? That would be odd, because this is no stronger than the original Newcomb problem and does not seem to contain any logical impossibilities.
Can he not compute the strategy, S, with the property “that at each moment, acting as S tells you to act—given (1) your beliefs about the universe at that point and (2) your intention of following S at all times—maximizes your net utility [over all time]?” That would be very odd, since you seem to believe a regular person can compute S. If you can do it, why not Omega? (NB, no, it doesn’t help to define an approximation of S and use that. If it’s rational, Omega will punish you for it. If it’s not, why are you doing it?)
Can he not compare your strategy to S, given that he knows the value of each? That seems odd, because a pushdown automaton could make the comparison. Do you require Omega to be weaker than a pushdown automaton?
No?
Then is it possible, maybe, that the problem is in the definition of S?
Well, for instance, he cannot make 1+1=3. And, if one defines rationality as actually winning then he cannot act in such a way that rational people lose. This is perfectly obvious; and, in case you have misunderstood what I wrote (as it looks like you have), that is the only thing I said that Omega cannot do.
In the discussion of strategy S, my claim was not about what Omega can do but about what you (a person attempting to implement such a strategy) can consistently include in your model of the universe. If you are an S-rational agent, then Omega may decide to screw you over, in which case you lose; that’s OK (as far as the notion of rationality goes; it’s too bad for you) because S doesn’t purport to guarantee that you don’t lose.
What S does purport to do is to arrange that, in so far as the universe obeys your (incomplete, probabilistic, …) model of it, you win on average. Omega’s malfeasance is only a problem for this if it’s included in your model. Which it can’t be. Hence:
what your example shows [...] is that you can’t consistently expect Omega to act in a way that falsifies your beliefs and/or invalidates your strategies for acting.
(Actually, I think that’s not quite right. You could probably consistently expect that, provided your expectations about how he’s going to to it were vague enough.)
I did not claim, nor do I believe, that a regular person can compute a perfectly rational strategy in the sense I described. Nor do I believe that a regular person can play chess without making any mistakes. None the less, there is such a thing as playing chess well; and there is such a thing as being (imperfectly, but better than one might be) rational. Even with a definition of the sort Eliezer likes.
If one defines rationality in some way that isn’t about winning, your example shows that rationalists-in-such-a-sense might not win.
If one defines rationality as actually winning, your example shows that there are things that even Omega cannot do because they involve logical contradiction.
If one defines rationality as something like “expected winning given one’s model of the universe” (for quibbles, see below), your example shows that you can’t coherently carry around a model of the universe that includes a superbeing who deliberately acts so as to invalidate that model.
I find all three of these things rather unsurprising.
The traditional form of Newcomb’s problem doesn’t involve a superbeing deliberately acting so as to invalidate your model of the universe. That seems like a big enough difference from your version to invalidate inferences of the form “there’s no such thing as acting rationally in grobstein’s version of Newcomb’s problem; therefore it doesn’t make sense to use any version of Newcomb’s problem in forming one’s ideas about what constitutes acting rationally”.
I think the third definition is pretty much what Eliezer is getting at when he declares that rationalists/rationality should win. Tightening it up a bit, I think we get something like this: rationality is a strategy S such that at each moment, acting as S tells you to act—given (1) your beliefs about the universe at that point and (2) your intention of following S at all times—maximizes your net utility (calculated in whatever way you prefer; that is mostly not a question of rationality). This isn’t quite a definition, because there might turn out to be multiple such strategies, especially for people whose initial beliefs about the universe are sufficiently crazy. But if you add some condition to the effect that S and your initial beliefs shouldn’t be too unlike what’s generally considered (respectively) rational and right now, there might well be a unique solution to the equations. And it seems to me that what your example shows about this definition is simply that you can’t consistently expect Omega to act in a way that falsifies your beliefs and/or invalidates your strategies for acting. Which is (to me) not surprising, and not a reason for defining rationality in this sort of way.
What is it, pray tell, that Omega cannot do?
Can he not scan your brain and determine what strategy you are following? That would be odd, because this is no stronger than the original Newcomb problem and does not seem to contain any logical impossibilities.
Can he not compute the strategy, S, with the property “that at each moment, acting as S tells you to act—given (1) your beliefs about the universe at that point and (2) your intention of following S at all times—maximizes your net utility [over all time]?” That would be very odd, since you seem to believe a regular person can compute S. If you can do it, why not Omega? (NB, no, it doesn’t help to define an approximation of S and use that. If it’s rational, Omega will punish you for it. If it’s not, why are you doing it?)
Can he not compare your strategy to S, given that he knows the value of each? That seems odd, because a pushdown automaton could make the comparison. Do you require Omega to be weaker than a pushdown automaton?
No?
Then is it possible, maybe, that the problem is in the definition of S?
Well, for instance, he cannot make 1+1=3. And, if one defines rationality as actually winning then he cannot act in such a way that rational people lose. This is perfectly obvious; and, in case you have misunderstood what I wrote (as it looks like you have), that is the only thing I said that Omega cannot do.
In the discussion of strategy S, my claim was not about what Omega can do but about what you (a person attempting to implement such a strategy) can consistently include in your model of the universe. If you are an S-rational agent, then Omega may decide to screw you over, in which case you lose; that’s OK (as far as the notion of rationality goes; it’s too bad for you) because S doesn’t purport to guarantee that you don’t lose.
What S does purport to do is to arrange that, in so far as the universe obeys your (incomplete, probabilistic, …) model of it, you win on average. Omega’s malfeasance is only a problem for this if it’s included in your model. Which it can’t be. Hence:
(Actually, I think that’s not quite right. You could probably consistently expect that, provided your expectations about how he’s going to to it were vague enough.)
I did not claim, nor do I believe, that a regular person can compute a perfectly rational strategy in the sense I described. Nor do I believe that a regular person can play chess without making any mistakes. None the less, there is such a thing as playing chess well; and there is such a thing as being (imperfectly, but better than one might be) rational. Even with a definition of the sort Eliezer likes.