I don’t think I buy this for Newcomb-like problems. Consider Omega who says, “There will be $1M in Box B IFF you are irrational.”
Rationality as winning is probably subject to a whole family of Russell’s-Paradox-type problems like that. I suppose I’m not sure there’s a better notion of rationality.
What you give is far harder than a Newcomb-like problem. In Newcomb-like problems, Omega rewards your decisions, he isn’t looking at how you reach them. This leaves you free to optimize those decisions.
What you give is far harder than a Newcomb-like problem. In Newcomb-like problems, Omega rewards your decisions, he isn’t looking at how you reach them.
You misunderstand. In my variant, Omega is also not looking at how you reach your decision. Rather, he is looking at you beforehand—“scanning your brain”, if you will—and evaluating the kind of person you are (i.e., how you “would” behave). This, along with the choice you make, determines your later reward.
In the classical problem, (unless you just assume backwards causation,) what Omega is doing is assessing the kind of person you are before you’ve physically indicated your choice. You’re rewarded IFF you’re the kind of person who would choose only box B.
My variant is exactly symmetrical: he assesses whether you are the kind of person who is rational, and responds as I outlined.
We have such an Omega: we just refer to it differently.
After all, we are used to treating our genes and our environments as definite influences on our ability to Win. Taller people tend to make more money; Omega says “there will be $1mil in box B if you have alleles for height.”
If Omega makes decisions based on properties of the agent, and not on the decisions either made or predicted to be made by the agent, then Omega is no different from, well, a lot of the world.
Rationality, then, might be better redefined under these observations as “making the decisions that Win whenever such decisions actually affect one’s probability of Winning,” though I prefer Eliezer’s more general rules plus the tacit understanding that we are only including situations where decisions make a difference.
(though I don’t see how you identify any distinction between “properties of the agent” and “decisions . . . predicted to be made by the agent” or why you care about it).
I’ll go further and say this distinction doesn’t matter unless you assume that Newcomb’s problem is a time paradox or some other kind of backwards causation.
Yes, all well and good (though I don’t see how you identify any distinction between “properties of the agent” and “decisions . . . predicted to be made by the agent” or why you care about it). My point is that a concept of rationality-as-winning can’t have a definite extension say across the domain of agents, because of the existence of Russell’s-Paradox problems like the one I identified.
This is perfectly robust to the point that weird and seemingly arbitrary properties are rewarded by the game known as the universe. Your proposed redefinition may actually disagree with EY’s theory of Newcomb’s problem. After all, your decision can’t empty box B, since the contents of box B are determinate by the time you make your decision.
After all, your decision can’t empty box B, since the contents of box B are determinate by the time you make your decision.
Hello. My name is Omega. Until recently I went around claiming to be all-knowing/psychic/whatever, but now I understand lying is Wrong, so I’m turning over a new leaf. I’d like to offer you a game.
Here are two boxes. Box A contains $1,000, box B contains $1,000,000. Both boxes are covered by touch-sensitive layer. If you choose box B only (please signal that by touching box B), it will send out a radio signal to box A, which will promptly disintegrate. If you choose both boxes (please signal that by touching box A first), a radio signal will be sent out to box B, which will disintegrate it’s content, so opening it will reveal an empty box.
(I got the disintegrating technology from the wreck of a UFO that crashed into my barn, but that’s not relevant here.)
I’m afraid, if I or my gadgets detect any attempt to temper with the operation of my boxes, I will be forced to disqualify you.
In case there is doubt, this is the same game I used to offer back in my deceitful days. The difference is, now the player knows the rules are enforced by cold hard electronics, so there’s no temptation to try and outsmart anybody.
Yes, you are changing the hypo. Your Omega dummy says that it is the same game as Newcomb’s problem, but it’s not. As VN notes, it may be equivalent to the version of Newcomb’s problem that assumes time travel, but this is not the classical (or an interesting) statement of the problem.
What is your point? You seem to be giving a metaphor for solving the problem by imagining that your action has a direct consequence of changing the past (and as a result, contents of the box in the present). More about that in this comment.
How Omega decides what to predict or what makes it’s stated condition for B (aka. result of “prediction”) come true, is not relevant. Ignoring the data that says it’s always/almost always correct, however, seems … not right. Any decision must be made with the understanding that Omega is most likely to predict it. You can’t outsmart it by failing to update it’s expected state of mind in the last second. The moment you decide to two-box is the moment Omega predicted, when it chose to empty box B.
Consider this:
Andy: “Sure, one box seems like the good choice, because Omega would take the million away otherwise. OK. … Now that the boxes are in front of me, I’m thinking I should take both. Because, you know, two is better than one. And it’s already decided, so my choice won’t change anything. Both boxes.”
Barry: “Sure, one box seems like the good choice, because Omega would take the million away otherwise. OK. … Now that the boxes are in front of me, I’m thinking I should take both. Because, you know, two is better than one. Of course the outcome still depends on what Omega predicted. Say I choose both boxes. So if Omega’s prediction is correct this time, I will find an empty B. But maybe Omega was wrong THIS time. Sure, and maybe THIS time I will also win the lottery. How it would have known is not relevant. The fact that O already acted on it’s prediction doesn’t make it more likely to be wrong. Really, what is the dilemma here? One box.”
Ok, I don’t expect that I’m the first person to say all this. But then, I wouldn’t have expected anybody to two-box, either.
I can see the relation to Newcomb—this is also a weird counterfactual that will never happen. I haven’t deliberately touched a hot stove in my adult life, and don’t expect to. I certainly won’t get to 99 times.
If one defines rationality in some way that isn’t about winning, your example shows that rationalists-in-such-a-sense might not win.
If one defines rationality as actually winning, your example shows that there are things that even Omega cannot do because they involve logical contradiction.
If one defines rationality as something like “expected winning given one’s model of the universe” (for quibbles, see below), your example shows that you can’t coherently carry around a model of the universe that includes a superbeing who deliberately acts so as to invalidate that model.
I find all three of these things rather unsurprising.
The traditional form of Newcomb’s problem doesn’t involve a superbeing deliberately acting so as to invalidate your model of the universe. That seems like a big enough difference from your version to invalidate inferences of the form “there’s no such thing as acting rationally in grobstein’s version of Newcomb’s problem; therefore it doesn’t make sense to use any version of Newcomb’s problem in forming one’s ideas about what constitutes acting rationally”.
I think the third definition is pretty much what Eliezer is getting at when he declares that rationalists/rationality should win. Tightening it up a bit, I think we get something like this: rationality is a strategy S such that at each moment, acting as S tells you to act—given (1) your beliefs about the universe at that point and (2) your intention of following S at all times—maximizes your net utility (calculated in whatever way you prefer; that is mostly not a question of rationality). This isn’t quite a definition, because there might turn out to be multiple such strategies, especially for people whose initial beliefs about the universe are sufficiently crazy. But if you add some condition to the effect that S and your initial beliefs shouldn’t be too unlike what’s generally considered (respectively) rational and right now, there might well be a unique solution to the equations. And it seems to me that what your example shows about this definition is simply that you can’t consistently expect Omega to act in a way that falsifies your beliefs and/or invalidates your strategies for acting. Which is (to me) not surprising, and not a reason for defining rationality in this sort of way.
Can he not scan your brain and determine what strategy you are following? That would be odd, because this is no stronger than the original Newcomb problem and does not seem to contain any logical impossibilities.
Can he not compute the strategy, S, with the property “that at each moment, acting as S tells you to act—given (1) your beliefs about the universe at that point and (2) your intention of following S at all times—maximizes your net utility [over all time]?” That would be very odd, since you seem to believe a regular person can compute S. If you can do it, why not Omega? (NB, no, it doesn’t help to define an approximation of S and use that. If it’s rational, Omega will punish you for it. If it’s not, why are you doing it?)
Can he not compare your strategy to S, given that he knows the value of each? That seems odd, because a pushdown automaton could make the comparison. Do you require Omega to be weaker than a pushdown automaton?
No?
Then is it possible, maybe, that the problem is in the definition of S?
Well, for instance, he cannot make 1+1=3. And, if one defines rationality as actually winning then he cannot act in such a way that rational people lose. This is perfectly obvious; and, in case you have misunderstood what I wrote (as it looks like you have), that is the only thing I said that Omega cannot do.
In the discussion of strategy S, my claim was not about what Omega can do but about what you (a person attempting to implement such a strategy) can consistently include in your model of the universe. If you are an S-rational agent, then Omega may decide to screw you over, in which case you lose; that’s OK (as far as the notion of rationality goes; it’s too bad for you) because S doesn’t purport to guarantee that you don’t lose.
What S does purport to do is to arrange that, in so far as the universe obeys your (incomplete, probabilistic, …) model of it, you win on average. Omega’s malfeasance is only a problem for this if it’s included in your model. Which it can’t be. Hence:
what your example shows [...] is that you can’t consistently expect Omega to act in a way that falsifies your beliefs and/or invalidates your strategies for acting.
(Actually, I think that’s not quite right. You could probably consistently expect that, provided your expectations about how he’s going to to it were vague enough.)
I did not claim, nor do I believe, that a regular person can compute a perfectly rational strategy in the sense I described. Nor do I believe that a regular person can play chess without making any mistakes. None the less, there is such a thing as playing chess well; and there is such a thing as being (imperfectly, but better than one might be) rational. Even with a definition of the sort Eliezer likes.
I don’t think I buy this for Newcomb-like problems. Consider Omega who says, “There will be $1M in Box B IFF you are irrational.”
Rationality as winning is probably subject to a whole family of Russell’s-Paradox-type problems like that. I suppose I’m not sure there’s a better notion of rationality.
What you give is far harder than a Newcomb-like problem. In Newcomb-like problems, Omega rewards your decisions, he isn’t looking at how you reach them. This leaves you free to optimize those decisions.
You misunderstand. In my variant, Omega is also not looking at how you reach your decision. Rather, he is looking at you beforehand—“scanning your brain”, if you will—and evaluating the kind of person you are (i.e., how you “would” behave). This, along with the choice you make, determines your later reward.
In the classical problem, (unless you just assume backwards causation,) what Omega is doing is assessing the kind of person you are before you’ve physically indicated your choice. You’re rewarded IFF you’re the kind of person who would choose only box B.
My variant is exactly symmetrical: he assesses whether you are the kind of person who is rational, and responds as I outlined.
We have such an Omega: we just refer to it differently.
After all, we are used to treating our genes and our environments as definite influences on our ability to Win. Taller people tend to make more money; Omega says “there will be $1mil in box B if you have alleles for height.”
If Omega makes decisions based on properties of the agent, and not on the decisions either made or predicted to be made by the agent, then Omega is no different from, well, a lot of the world.
Rationality, then, might be better redefined under these observations as “making the decisions that Win whenever such decisions actually affect one’s probability of Winning,” though I prefer Eliezer’s more general rules plus the tacit understanding that we are only including situations where decisions make a difference.
Quoting myself:
I’ll go further and say this distinction doesn’t matter unless you assume that Newcomb’s problem is a time paradox or some other kind of backwards causation.
This is all tangential, though, I think.
Yes, all well and good (though I don’t see how you identify any distinction between “properties of the agent” and “decisions . . . predicted to be made by the agent” or why you care about it). My point is that a concept of rationality-as-winning can’t have a definite extension say across the domain of agents, because of the existence of Russell’s-Paradox problems like the one I identified.
This is perfectly robust to the point that weird and seemingly arbitrary properties are rewarded by the game known as the universe. Your proposed redefinition may actually disagree with EY’s theory of Newcomb’s problem. After all, your decision can’t empty box B, since the contents of box B are determinate by the time you make your decision.
Hello. My name is Omega. Until recently I went around claiming to be all-knowing/psychic/whatever, but now I understand lying is Wrong, so I’m turning over a new leaf. I’d like to offer you a game.
Here are two boxes. Box A contains $1,000, box B contains $1,000,000. Both boxes are covered by touch-sensitive layer. If you choose box B only (please signal that by touching box B), it will send out a radio signal to box A, which will promptly disintegrate. If you choose both boxes (please signal that by touching box A first), a radio signal will be sent out to box B, which will disintegrate it’s content, so opening it will reveal an empty box.
(I got the disintegrating technology from the wreck of a UFO that crashed into my barn, but that’s not relevant here.)
I’m afraid, if I or my gadgets detect any attempt to temper with the operation of my boxes, I will be forced to disqualify you.
In case there is doubt, this is the same game I used to offer back in my deceitful days. The difference is, now the player knows the rules are enforced by cold hard electronics, so there’s no temptation to try and outsmart anybody.
So, what will it be?
Yes, you are changing the hypo. Your Omega dummy says that it is the same game as Newcomb’s problem, but it’s not. As VN notes, it may be equivalent to the version of Newcomb’s problem that assumes time travel, but this is not the classical (or an interesting) statement of the problem.
What is your point? You seem to be giving a metaphor for solving the problem by imagining that your action has a direct consequence of changing the past (and as a result, contents of the box in the present). More about that in this comment.
Naive argument coming up.
How Omega decides what to predict or what makes it’s stated condition for B (aka. result of “prediction”) come true, is not relevant. Ignoring the data that says it’s always/almost always correct, however, seems … not right. Any decision must be made with the understanding that Omega is most likely to predict it. You can’t outsmart it by failing to update it’s expected state of mind in the last second. The moment you decide to two-box is the moment Omega predicted, when it chose to empty box B.
Consider this:
Andy: “Sure, one box seems like the good choice, because Omega would take the million away otherwise. OK. … Now that the boxes are in front of me, I’m thinking I should take both. Because, you know, two is better than one. And it’s already decided, so my choice won’t change anything. Both boxes.”
Barry: “Sure, one box seems like the good choice, because Omega would take the million away otherwise. OK. … Now that the boxes are in front of me, I’m thinking I should take both. Because, you know, two is better than one. Of course the outcome still depends on what Omega predicted. Say I choose both boxes. So if Omega’s prediction is correct this time, I will find an empty B. But maybe Omega was wrong THIS time. Sure, and maybe THIS time I will also win the lottery. How it would have known is not relevant. The fact that O already acted on it’s prediction doesn’t make it more likely to be wrong. Really, what is the dilemma here? One box.”
Ok, I don’t expect that I’m the first person to say all this. But then, I wouldn’t have expected anybody to two-box, either.
major said:
You’re not the only person to wonder this. Either I’m missing something, or two-boxers just fail at induction.
I have to wonder how two-boxers would do on the “Hot Stove Problem.”
In case you guys haven’t heard of such a major problem in philosophy, I will briefly explain the Hot Stove Problem:
You have touched a hot stove 100 times. 99 times you have been burned. Nothing has changed about the stove that you know about. Do you touch it again?
I can see the relation to Newcomb—this is also a weird counterfactual that will never happen. I haven’t deliberately touched a hot stove in my adult life, and don’t expect to. I certainly won’t get to 99 times.
If one defines rationality in some way that isn’t about winning, your example shows that rationalists-in-such-a-sense might not win.
If one defines rationality as actually winning, your example shows that there are things that even Omega cannot do because they involve logical contradiction.
If one defines rationality as something like “expected winning given one’s model of the universe” (for quibbles, see below), your example shows that you can’t coherently carry around a model of the universe that includes a superbeing who deliberately acts so as to invalidate that model.
I find all three of these things rather unsurprising.
The traditional form of Newcomb’s problem doesn’t involve a superbeing deliberately acting so as to invalidate your model of the universe. That seems like a big enough difference from your version to invalidate inferences of the form “there’s no such thing as acting rationally in grobstein’s version of Newcomb’s problem; therefore it doesn’t make sense to use any version of Newcomb’s problem in forming one’s ideas about what constitutes acting rationally”.
I think the third definition is pretty much what Eliezer is getting at when he declares that rationalists/rationality should win. Tightening it up a bit, I think we get something like this: rationality is a strategy S such that at each moment, acting as S tells you to act—given (1) your beliefs about the universe at that point and (2) your intention of following S at all times—maximizes your net utility (calculated in whatever way you prefer; that is mostly not a question of rationality). This isn’t quite a definition, because there might turn out to be multiple such strategies, especially for people whose initial beliefs about the universe are sufficiently crazy. But if you add some condition to the effect that S and your initial beliefs shouldn’t be too unlike what’s generally considered (respectively) rational and right now, there might well be a unique solution to the equations. And it seems to me that what your example shows about this definition is simply that you can’t consistently expect Omega to act in a way that falsifies your beliefs and/or invalidates your strategies for acting. Which is (to me) not surprising, and not a reason for defining rationality in this sort of way.
What is it, pray tell, that Omega cannot do?
Can he not scan your brain and determine what strategy you are following? That would be odd, because this is no stronger than the original Newcomb problem and does not seem to contain any logical impossibilities.
Can he not compute the strategy, S, with the property “that at each moment, acting as S tells you to act—given (1) your beliefs about the universe at that point and (2) your intention of following S at all times—maximizes your net utility [over all time]?” That would be very odd, since you seem to believe a regular person can compute S. If you can do it, why not Omega? (NB, no, it doesn’t help to define an approximation of S and use that. If it’s rational, Omega will punish you for it. If it’s not, why are you doing it?)
Can he not compare your strategy to S, given that he knows the value of each? That seems odd, because a pushdown automaton could make the comparison. Do you require Omega to be weaker than a pushdown automaton?
No?
Then is it possible, maybe, that the problem is in the definition of S?
Well, for instance, he cannot make 1+1=3. And, if one defines rationality as actually winning then he cannot act in such a way that rational people lose. This is perfectly obvious; and, in case you have misunderstood what I wrote (as it looks like you have), that is the only thing I said that Omega cannot do.
In the discussion of strategy S, my claim was not about what Omega can do but about what you (a person attempting to implement such a strategy) can consistently include in your model of the universe. If you are an S-rational agent, then Omega may decide to screw you over, in which case you lose; that’s OK (as far as the notion of rationality goes; it’s too bad for you) because S doesn’t purport to guarantee that you don’t lose.
What S does purport to do is to arrange that, in so far as the universe obeys your (incomplete, probabilistic, …) model of it, you win on average. Omega’s malfeasance is only a problem for this if it’s included in your model. Which it can’t be. Hence:
(Actually, I think that’s not quite right. You could probably consistently expect that, provided your expectations about how he’s going to to it were vague enough.)
I did not claim, nor do I believe, that a regular person can compute a perfectly rational strategy in the sense I described. Nor do I believe that a regular person can play chess without making any mistakes. None the less, there is such a thing as playing chess well; and there is such a thing as being (imperfectly, but better than one might be) rational. Even with a definition of the sort Eliezer likes.