For example, in a Newcomb-type problem, suppose I decide to resolve the question of one box or two by flipping a coin? Unless I am supposed to believe that Omega can foretell the results of future coin flips, I think the scenario collapses. Has anyone written anything on LW about responding to Omega by randomizing?
Yes, back when we discussed Newcomblike problems frequently I more or less used a form letter to reply to that objection. Any useful treatment of Newcomblike problems will specify explicitly or implicitly how Omega will handle (quantum) randomness if it is allowed. The obvious response for Omega is to either give you nothing (or maybe a grenade!) for being a smart ass or, more elegantly, handle the reward given in commensurate manner to the probabilities. If probabilistic decisions are to be allowed then an Omega that can handle probabilistic decisions quite clearly needs to be supplied.
Thanks for posting. Your analysis is an improvement over the LW conventional wisdom, but you still doesn’t get it right, where right, to me, means the way it is analyzed by the guys who won all those Nobel prizes in economics.
I downvoted the parent. How on earth is Perplexed comparing LW conventional wisdom to that of Nobel prize winning economists when he thinks coin tossing is a big deal?
Any useful treatment of Newcomblike problems will specify explicitly or implicitly how Omega will handle (quantum) randomness.
At the risk of appearing stupid, I have to ask: exactly what is a “useful treatment of Newcomb-like problems” used for?
So far, the only effect that all the Omega-talk has had on me is to make me honestly suspect that you guys must be into some kind of mind-over-matter quantum woo.
Seriously, Omega is not just counterfactual, he is impossible. Why do you guys keep asking us to believe so many impossible things before breakfast? Jaynes says not to include impossible propositions among the conditions in a conditional probability. Bad things happen if you do. Impossible things need to have zero-probability priors. Omega just has no business hanging around with honest Bayesians.
When I read that you all are searching for improved decision theories that “solve” the one-shot prisoner’s dilemma and the one-shot Parfit hitchhiker, I just cringe. Surely you shouldn’t change the standard, well-established, and correct decision theories. If you don’t like the standard solutions, you should instead revise the problems from unrealistic one-shots to more realistic repeated games or perhaps even more realistic games with observers—observers who may play games with you in the future.
In every case I have seen so far where Eliezer has denigrated the standard game solution because it fails to win, he has been analyzing a game involving a physically and philosophically impossible fictional situation.
Let me ask the question this way: What evidence do you have that the standard solution to the one-shot PD can be improved upon without creating losses elsewhere? My impression is that you are being driven by wishful thinking and misguided intuition.
Here’s another way of looking at the situation that may or may not be helpful. Suppose I ask you, right here and now, what you’d do in the hypothetical future Parfit’s Hitchhiker scenario if your opponent was a regular human with Internet access. You have several options:
Answer truthfully that you’d pay $100, thus proving that you don’t subscribe to CDT or EDT. (This is the alternative I would choose.)
Answer that you’d refuse to pay. Now you’ve created evidence on the Internet, and if/when you face the scenario in real life, the driver will Google your name, check the comments on LW and leave you in the desert to die. (Assume the least convenient possible world where you can’t change or delete your answer once it’s posted.)
Answer that you’d pay up, but secretly plan to refuse. This means you’d be lying to us here in the comments—surely not a very nice thing to do. But if you subscribe to CDT with respect to utterances as well as actions, this is the alternative you’re forced to choose. (Which may or may not make you uneasy about CDT.)
Answer that you’d pay up, but secretly plan to refuse. This means you’d be lying to us here in the comments—surely not a very nice thing to do. But if you subscribe to CDT with respect to utterances as well as actions, this is the alternative you’re forced to choose. (Which may or may not make you uneasy about CDT.)
What makes me uneasy is the assumption I wouldn’t want to pay $100 to somebody who rescued me from the desert. Given that, lying to people whom I don’t really know should be a piece of cake!
I would of course choose option #1, adding that, due to an affliction giving me a trembling hand, I tend to get stranded in the desert and the like a lot and hence that I would appreciate it if he would spread the story of my honesty among other drivers. I might also promise to keep secret the fact of his own credulity in this case, should he ask me to. :)
I understand quite well that the best and simplest way to appear honest is to actually be honest. And also that, as a practical matter, you never really know who might observe your selfish actions and how that might hurt you in the future. But these prudential considerations can already be incorporated into received decision theory (which, incidentally, I don’t think matches up with either CDT or EDT—at least as those acronyms seem to be understood here.) We don’t seem to need TDT and UDT to somehow glue them in to the foundations.
Hmmm. Is EY perhaps worried that an AI might need need even stronger inducements toward honesty? Maybe it would, but I don’t see how you solve the problem by endowing the AI with a flawed decision theory.
So far, the only effect that all the Omega-talk has had on me is to make me honestly suspect that you guys must be into some kind of mind-over-matter quantum woo.
...What?
Also, it doesn’t matter if he’s impossible. He’s an easy way to tack on arbitrary rules to hypotheticals without overly tortured explanations, because people are used to getting arbitrary rules from powerful agents.
It’s also impossible for a perfectly Absent Minded Driver to come to one of only two possible intersections with 3 destinations with known payoffs and no other choices. To say nothing of the impossibly horrible safety practices of our nation’s hypothetical train system.
Are you sure? I’m not objecting to the arbitrary payoffs or complaining because he doesn’t seem to be maximizing his own utility. I’m objecting to his ability to predict my actions. Give me a scenario which doesn’t require me to assign a non-zero prior to woo and in which a revisionist decision theory wins. If you can’t, then your “improved” decision theory is no better than woo itself.
Regarding the Absent Minded Driver, I didn’t recognize the reference. Googling, I find a .pdf by one of my guys (Nobelist Robert Aumann) and an LW article by Wei-Dai. Cool, but since it is already way past my bedtime, I will have to read them in the morning and get back to you.
I’m objecting to his ability to predict my actions. Give me a scenario which doesn’t require me to assign a non-zero prior to woo
The only ‘woo’ here seems to be your belief that your actions are not predictable (even in principle!). Even I can predict your actions within some tolerances, and we do not need to posit that I am a superintelligence! Examples: you will not hang yourself to death within the next five minutes, and you will ever make another comment on Less Wrong.
You need a little more context/priming or to make the joke longer for anyone to catch this. Or you need to embed it in a more substantive and sensible reply. Otherwise it will hardly ever work.
I’m objecting to his ability to predict my actions.
Why?
What about you is fundamentally logically impossible to predict?
Do you not find that you often predict the actions of others? (ie. giving them gifts that you know they’ll like)
And that others predict your reactions? (ie. choosing not to give you spider-themed horror movies if you’re arachnophobic)
Give me a scenario which doesn’t require me to assign a non-zero prior to woo and in which a revisionist decision theory wins.
Omega is a perfect super-intelligence, existing in a computer simulation like universe that can be modeled by a set of physical laws and a very long string of random numbers. Omega knows the laws and the numbers.
Ok, I’ve read the paper(most of it) and Wei-Dai’s article now. Two points.
In a sense, I understand how you might think that the Absent Minded Driver is no less contrived and unrealistic than Newcomb’s Paradox. Maybe different people have different intuitions as to what toy examples are informative and which are misleading. Someone else (on this thread?) responded to me recently with the example of frictionless pulleys and the like from physics. All I can tell you is that my intuition tells me that the AMD, the PD, frictionless pulleys,and even Parfit’s Hitchhiker all strike me as admirable teaching tools, whereas Newcomb problems and the old questions of irrestable force vs immovable object in physics are simply wrong problems which can only create confusion.
Reading Wei-Dai’s snarking about how the LW approach to decision theory (with zero published papers to date) is so superior to the confusion in which mere misguided Nobel laureates struggle—well, I almost threw up. It is extremely doubtful that I will continue posting here for long.
It wasn’t meant to be a snark. I was genuinely trying to figure out how the “LW approach” might be superior, because otherwise the most likely explanation is that we’re all deluded in thinking that we’re making progress. I’d be happy to take any suggestions on how I could have reworded my post so that it sounded less like a snark.
Wei-Dai wrote a post entitled The Absent-Minded Driver which I labeled “snarky”. Moreover, I suggested that the snarkiness was so bad as to be nauseating, so as to drive reasonable people to flee in horror from LW and SAIA. I here attempt to defend these rather startling opinions. Here is what Wei-Dai wrote that offended me:
This post examines an attempt by professional decision theorists to treat an example of time inconsistency, and asks why they failed to reach the solution (i.e., TDT/UDT) that this community has more or less converged upon. (Another aim is to introduce this example, which some of us may not be familiar with.) Before I begin, I should note that I don’t think “people are crazy, the world is mad” (as Eliezer puts it) is a good explanation. Maybe people are crazy, but unless we can understand how and why people are crazy (or to put it more diplomatically, “make mistakes”), how can we know that we’re not being crazy in the same way or making the same kind of mistakes?
The paper that Wei-Dai reviews is “The Absent-Minded Driver” by Robert J. Aumann, Sergiu Hart, and Motty Perry. Wei-Dai points out, rather condescendingly:
(Notice that the authors of this paper worked for a place called Center for the Study of Rationality, and one of them won a Nobel Prize in Economics for his work on game theory. I really don’t think we want to call these people “crazy”.)
Wei-Dai then proceeds to give a competent description of the problem and the standard “planning-optimality” solution of the problem. Next comes a description of an alternative seductive-but-wrong solution by Piccione and Rubinstein. I should point that everyone—P&R, Aumann, Hart, and Perry, Wei-Dai, me, and hopefully you who look into this—realizes that the alternative P&R solution is wrong. It gets the wrong result. It doesn’t win. The only problem is explaining exactly where the analysis leading to that solution went astray, and in explaining how it might be modified so as to go right. Making this analysis was, as I see it, the whole point of both papers—P&R and Aumann et al. Wei-Dai describes some characteristics of Aumann et al’s corrected version of the alternate solution. Then he (?) goes horribly astray:
In problems like this one, UDT is essentially equivalent to planning-optimality. So why did the authors propose and argue for action-optimality despite its downsides …, instead of the alternative solution of simply remembering or recomputing the planning-optimal decision at each intersection and carrying it out?
But, as anyone who reads the paper carefully should see, they weren’t arguing for action-optimality as the solution. They never abandoned planning optimality. Their point is that if you insist on reasoning in this way, (and Seldin’s notion of “subgame perfection” suggests some reasons why you might!) then the algorithm they call “action-optimality” is the way to go about it.
But Wei-Dai doesn’t get this. Instead we get this analysis of how these brilliant people just haven’t had the educational advantages that LW folks have:
Well, the authors don’t say (they never bothered to argue against it), but I’m going to venture some guesses:
That solution is too simple and obvious, and you can’t publish a paper arguing for it.
It disregards “the probability of being at X”, which intuitively ought to play a role.
The authors were trying to figure out what is rational for human beings, and that solution seems too alien for us to accept and/or put into practice.
The authors were not thinking in terms of an AI, which can modify itself to use whatever decision theory it wants to.
Aumann is known for his work in game theory. The action-optimality solution looks particularly game-theory like, and perhaps appeared more natural than it really is because of his specialized knowledge base.
The authors were trying to solve one particular case of time inconsistency. They didn’t have all known instances of time/dynamic/reflective inconsistencies/paradoxes/puzzles laid out in front of them, to be solved in one fell swoop.
Taken together, these guesses perhaps suffice to explain the behavior of these professional rationalists, without needing to hypothesize that they are “crazy”. Indeed, many of us are probably still not fully convinced by UDT for one or more of the above reasons.
Let me just point out that the reason it is true that “they never argued against it” is that they had already argued for it. Check out the implications of their footnote #4!
Ok, those are the facts, as I see them. Was Wei-Dai snarky? I suppose it depends on how you define snarkiness. Taboo “snarky”. I think that he was overbearingly condescending without the slightest real reason for thinking himself superior. “Snarky” may not be the best one-word encapsulation of that attitude, but it is the one I chose. I am unapologetic. Wei-Dai somehow came to believe himself better able to see the truth than a Nobel laureate in the Nobel laureate’s field. It is a mistake he would not have made had he simply read a textbook or taken a one-semester course in the field. But I’m coming to see it as a mistake made frequently by SIAI insiders.
Let me point out that the problem of forgetful agents may seem artificial, but it is actually extremely important. An agent with perfect recall playing the iterated PD, knowing that it is to be repeated exactly 100 times, should rationally choose to defect. On the other hand, if he cannot remember how many iterations remain to be played, and knows that the other player cannot remember either, should cooperate by playing Tit-for-Tat or something similar.
Well, that is my considered response on “snarkiness”. I still have to respond on some other points, and I suspect that, upon consideration, I am going to have to eat some crow. But I’m not backing down on this narrow point. Wei-Dai blew it in interpreting Aumann’s paper. (And also, other people who know some game theory should read the paper and savor the implications of footnote #4. It is totally cool).
The paper that Wei-Dai reviews is “The Absent-Minded Driver” by Robert J. Aumann, Sergiu Hart, and Motty Perry. Wei-Dai points out, rather condescendingly:
(Notice that the authors of this paper worked for a place called Center for the Study of Rationality, and one of them won a Nobel Prize in Economics for his work on game theory. I really don’t think we want to call these people “crazy”.)
How is Wei Dai being condescending there? He’s pointing out how weak it is to dismiss people with these credentials by just calling them crazy. ETA: In other words, it’s an admonishment directed at LWers.
I’m sure it would be Wei-Dai’s read as well. The thing is, if Wei-Dai had not mistakenly come to the conclusion that the authors are wrong and not as enlightened as LWers, that admonishment would not be necessary. I’m not saying he condescends to LWers. I say he condescends to the rest of the world, particularly game theorists.
No. Not at all. It is because he disagreed through the wrong channels, and then proceeded to propose rather insulting hypotheses as to why they had gotten it wrong.
Just read that list of possible reasons! And there are people here arguing that “of course we want to analyze the cause of mistakes”. Sheesh. No wonder folks here are so in love with Evolutionary Psychology.
Ok, I’m probably going to get downvoted to hell because of that last paragraph. And,
you know what, that downvoting impulse due to that paragraph pretty much makes my case for why Wei Dai was wrong to do what he did. Think about it.
Ok, I’m probably going to get downvoted to hell because of that last paragraph. And, you know what, that downvoting impulse due to that paragraph pretty much makes my case for why Wei Dai was wrong to do what he did. Think about it.
Interestingly enough I think that it is this paragraph that people will downvote, and not the one above. Mind you, the premise in “No wonder folks here are so in love with Evolutionary Psychology.” does seem so incredibly backward that I almost laughed.
No. Not at all. It is because he disagreed through the wrong channels, and then proceeded to propose rather insulting hypotheses as to why they had gotten it wrong.
I can understand your explanation here. Without agreeing with it myself I can see how it follows from your premises.
Are you saying that you read him differently, and that he would somehow be misinterpreting himself?
The thing is, if Wei-Dai had not mistakenly come to the conclusion that the authors are wrong and not as enlightened as LWers, that admonishment would not be necessary.
The admonishment is necessary if LWers are likely to wrongly dismiss Aumann et al. as “crazy”. In other words, to think that the admonishment is necessary is to think that LWers are too inclined to dismiss other people as crazy
I’m not saying he condescends to LWers. I say he condescends to the rest of the world, particularly game theorists.
I got that. Who said anything about condescending to LWers?
Are you saying that you read him differently, and that he would somehow be misinterpreting himself?
Huh?? Surely, you troll. I am saying that Wei-Dai’s read would likely be the same as yours: that he was not condescending; that he was in fact cautioning his readers against looking down on the poor misguided Nobelists who, after all, probably had good reasons for being so mistaken. There, but for the grace of EY, go we.
Condescension is a combination of content and context. When you isolated that quote as especially condescending, I thought that you read something within it that was condescending. I was confused, because the quote could just as well have come from a post arguing that LWers ought to believe that Aumann et al. are right.
It now looks like you and I read the intrinsic meaning of the quote in the same way. The question then is, does that quote, placed in context, somehow make the overall post more condescending than it already was? Wei had already said that his treatment of the AMD was better than that of Aumann et al.. He had already said that these prestigious researchers got it wrong. Do you agree that if this were true, if the experts got it wrong, then we ought to try to understand how that happened, and not just dismiss them as crazy?
Whatever condescension occurred, it occurred as soon as Wei said that he was right and Aumann et al. were wrong. How can drawing a rational inference from that belief make it more condescending?
In this light I can see where ‘condescension’ fits in. There is a difference between ‘descending to be with’ and just plain ‘being way above’. For example we could label “they are wrong” as arrogant, “they are wrong but we can empathise with them and understand their mistake” as condescending and “They are wrong, that’s the kind of person Nobel prizes go to these days?” as “contemptuous”—even though they all operate from the same “I consider myself above in this instance” premise. Wei’s paragraph could then be considered to be transferring weight from arrogance and contempt into condescension.
(I still disapprove of Perplexed’s implied criticism.)
Okay, I can see this distinction. I can see how, as a matter of social convention, “they are wrong but we should understand their mistake” could come across as more condescending than just “they are wrong”. But I really don’t like that convention. If an expert is wrong, we really do have an obligation to understand how that happened. Accepting that obligation shouldn’t be stigmatized as condescending. (Not that you implied otherwise.)
the question then is, does that quote, placed in context, somehow make the overall post more condescending than it already was?
“They are probably not crazy” strikes me as “damning with faint praise”. IMHO, it definitely raises the overall condescension level.
Whatever condescension occurred, it occurred as soon as Wei said that he was right and Aumann et al. were wrong.
No. Peons claim lords are wrong all the time. It is not even impolite, if you are willing to admit your mistake and withdraw your claim reasonably quickly.
Condescension starts when you attempt to “charitably” analyze the source of the error.
Do you agree that if this were true, if the experts got it wrong, then we ought to try to understand how that happened, and not just dismiss them as crazy?
Of course. But if I merely had good reason to believe they were wrong, then my most urgent next step would be to determine whether it were true that they got it wrong. I would begin by communicating with the experts, either privately or through the peer-reviewed literature, so as to get some feedback as to whether they were wrong or I was mistaken. If it does indeed turn out that they were wrong, I would let them take the first shot at explaining the causes of their mistake. I doubt that I would try to analyze the cause of the mistake myself unless I were a trained historian dealing with a mistake at least 50 years old. Or, if I did try (and I probably have), I would hope that someone would point out my presumption.
Preliminary notes: You can call me “Wei Dai” (that’s firstname lastname). “He” is ok. I have taken a graduate level course in game theory (where I got a 4.0 grade, in case you suspect that I coasted through it), and have Fudenberg and Tirole’s “Game Theory” and Joyce’s “Foundations of Causal Decision Theory” as two of the few physical books that I own.
Their point is that if you insist on reasoning in this way, (and Seldin’s notion of “subgame perfection” suggests some reasons why you might!) then the algorithm they call “action-optimality” is the way to go about it.
I can’t see where they made this point. At the top of Section 4, they say “How, then, should the driver reason at the action stage?” and go on directly to describe action-optimality. If they said something like “One possibility is to just recompute and apply the planning-optimal solution. But if you insist …” please point out where. See also page 108:
In our case, there is only one player, who acts at different times.
Because of his absent-mindedness, he had better coordinate his actions;
this coordination can take place only before he starts out}at the planning
stage. At that point, he should choose p*1 . If indeed he chose p*1 , there is
no problem. If by mistake he chose p*2 or p*3 , then that is what he should
do at the action stage. (If he chose something else, or nothing at all, then
at the action stage he will have some hard thinking to do.)
If Aumann et al. endorse using planning-optimality at the action stage, why would they say the driver has some hard thinking to do? Again, why not just recompute and apply the planning-optimal solution?
I also do not see how subgame perfection is relevant here. Can you explain?
Let me just point out that the reason it is true that “they never argued against it” is that they had already argued for it. Check out the implications of their footnote #4!
This footnote?
Formally, (p*, p*) is a symmetric Nash equilibrium in the (symmetric) game between ‘‘the driver at the current intersection’’ and ‘‘the driver at the other intersection’’ (the strategic form game with payoff functions h.)
Since p* is the action-optimal solution, they are pointing out the formal relationship between their notion of action-optimality and Nash equilibrium. How is this footnote an argument for “it” (it being “recomputing the planning-optimal decision at each intersection and carrying it out”)?
I have taken a graduate level course in game theory (where I got a 4.0 grade, in case you suspect that I coasted through it), and have Fudenberg and Tirole’s “Game Theory” and Joyce’s “Foundations of Causal Decision Theory” as two of the few physical books that I own.
Ok, so it is me who is convicted of condescending without having the background to justify it. :( FWIW I have never taken a course, though I have been reading in the subject for more than 45 years.
Relevance of Subgame perfection. Seldin suggested subgame perfection as a refinement of Nash equilibrium which requires that decisions that seemed rational at the planning stage ought to still seem rational at the action stage. This at least suggests that we might want to consider requiring “subgame perfection” even if we only have a single player making two successive decisions.
Relevance of Footnote #4. This points out that one way to think of problems where a single player makes a series of decisions is to pretend that the problem has a series of players making the decisions—one decision per player, but that these fictitious players are linked in that they all share the same payoffs (but not necessarily the same information). This is a standard “trick” in game theory, but the footnote points out that in this case, since both fictitious players have the same information (because of the absent-mindedness) the game between driver-version-1 and driver-version-2 is symmetric, and that is equivalent to the constraint p1 = p2.
Does Footnote #4 really amount to “they had already argued for [just recalculating the planning-optimal solution]”? Well, no it doesn’t really. I blew it in offering that as evidence. (Still think it is cool, though!)
Do they “argue for it” anywhere else? Yes, they do. Section 5, where they apply their methods to a slightly more complicated example, is an extended argument for the superiority of the planning-optimal solution to the action-optimal solutions. As they explain, there can be multiple action-optimal solutions, even if there is only one (correct) planning-optimal solution, and some of those action-optimal solutions are wrong *even though they appear to promise a higher expected payoff than does the planning optimal solution.
I can’t see where they made this point. At the top of Section 4, they say “How, then, should the driver reason at the action stage?” and go on directly to describe action-optimality. If they said something like “One possibility is to just recompute and apply the planning-optimal solution. But if you insist …” please point out where. See also page 108:
In our case, there is only one player, who acts at different times. Because of his absent-mindedness, he had better coordinate his actions; this coordination can take place only before he starts out at the planning stage. At that point, he should choose p1 . If indeed he chose p1 , there is no problem. If by mistake he chose p2 or p3 , then that is what he should do at the action stage. (If he chose something else, or nothing at all, then at the action stage he will have some hard thinking to do.)
If Aumann et al. endorse using planning-optimality at the action stage, why would they say the driver has some hard thinking to do? Again, why not just recompute and apply the planning-optimal solution?
I really don’t see why you are having so much trouble parsing this. “If indeed he chose p1 , there is no problem” is an endorsement of the correctness of the planning-optimal solution. The sentence dealing with p2 and p3 asserts that, if you mistakenly used p2 for your first decision, then you best follow-up is to remain consistent and use p2 for your remaining two choices. The paragraph you quote to make your case is one I might well choose myself to make my case.
Edit: There are some asterisks in variable names in the original paper which I was unable to make work with the italics rules on this site. So “p2” above should be read as “p 2″
It is a statement that the planning-optimal action is the correct one, but it’s not an endorsement that it is correct to use the planning-optimality algorithm to compute what to do when you are already at an intersection. Do you see the difference?
ETA (edited to add): According to my reading of that paragraph, what they actually endorse is to compute the planning-optimal action at START, remember that, then at each intersection, compute the set of action-optimal actions, and pick the element of the set that coincides with the planning-optimal action.
BTW, you can use “\” to escape special characters like “*” and “_”.
Thx for the escape character info. That really ought to be added to the editing help popup.
Yes, I see the difference. I claim that what they are saying here is that you need to do the planning-optimal calculation in order to find p*1 as the unique best solution (among the three solutions that the action-optimal method provides). Once you have this, you can use it at the first intersection. But at the other intersections, you have some choices: either recalculate the planning-optimal solution each time, or write down enough information so that you can recognize that p*1 is the solution you are already committed to among the three (in section 5) solutions returned by the action-optimality calculation.
ETA in response to your ETA. Yes they do. Good point. I’m pretty sure there are cases more complicated than this perfectly amnesiac driver where that would be the only correct policy. (ETA:To be more specific, cases where the planning-optimal solution is not a sequential equilibrium). But then I have no reason to think that UDT would yield the correct answer in those more complicated cases either.
I deleted my previous reply since it seems unnecessary given your ETA.
I’m pretty sure there are cases more complicated than this perfectly amnesiac driver where that would be the only correct policy. (ETA:To be more specific, cases where the planning-optimal solution is not a sequential equilibrium).
What would be the only correct policy? What I wrote after “According to my reading of that paragraph”? If so, I don’t understand your “cases where the planning-optimal solution is not a sequential equilibrium”. Please explain.
What would be the only correct policy? What I wrote after “According to my reading of that paragraph”?
Yes.
If so, I don’t understand your “cases where the planning-optimal solution is not a sequential equilibrium”. Please explain.
I would have thought it would be self explanatory.
It looks like I will need to construct and analyze examples slightly more complicated that the Absent Minded Driver. That may take a while. Questions before I start: Does UDT encompass game theory, or is it limited to analyzing single-player situations? Is UDT completely explained in your postings, or is it, like TDT, still in the process of being written up?
Questions before I start: Does UDT encompass game theory, or is it limited to analyzing single-player situations? Is UDT completely explained in your postings, or is it, like TDT, still in the process of being written up?
Wei has described a couple versions of UDT. His descriptions seemed to me to be mathematically rigorous. Based on Wei’s posts, I wrote this pdf, which gives just the definition of a UDT agent (as I understand it), without motivation or justification.
The difficulty with multiple agents looks like it will be very hard to get around within the UDT framework. UDT works essentially by passing the buck to an agent who is at the planning stage*. That planning-stage agent then performs a conventional expected-utility calculation.
But some scenarios seem best described by saying that there are multiple planning-stage agents. That means that UDT is subject to all of the usual difficulties that arise when you try to use expected utility alone in multiplayer games (e.g., prisoners dilemma). It’s just that these difficulties arise at the planning stage instead of at the action stage directly.
*Somewhat more accurately, the buck is passed to the UDT agent’s simulation of an agent who is at the planning stage.
What I meant was, what point were you trying to make with that statement? According to Aumann’s paper, every planning-optimal solution is also an action-optimal solution, so the decision procedure they endorse will end up picking the planning-optimal solution. (My complaint is just that it goes about it in an unnecessarily round-about way.) If theirs is a correct policy, then the policy of just recomputing the planning-optimal solution must also be correct. That seems to disprove your “only correct policy” claim. I thought your “sequential equilibrium” line was trying to preempt this argument, but I can’t see how.
Does UDT encompass game theory, or is it limited to analyzing single-player situations?
Pretty much single-player for now. A number of people are trying to extend the ideas to multi-player situations, but it looks really hard.
Is UDT completely explained in your postings, or is it, like TDT, still in the process of being written up?
No, it’s not being written up further. (Nesov is writing up some of his ideas, which are meant to be an advance over UDT.)
What I meant was, what point were you trying to make with that statement? According to Aumann’s paper, every planning-optimal solution is also an action-optimal solution, so the decision procedure they endorse will end up picking the planning-optimal solution.
My understanding of their paper has changed somewhat since we began this discussion. I now believe that repeating the planning-optimal analysis at every decision node is only guaranteed to give ideal results in simple cases like this one in which every decision point is in the same information set. In more complicated cases,
I can imagine that the policy of planning-optimal-for-the first-move, then action-optimal-thereafter might do better. I would need to construct an example to assert this with confidence.
(My complaint is just that it goes about it in an unnecessarily round-about way.) If theirs is a correct policy, then the policy of just recomputing the planning-optimal solution must also be correct.
In this simple example, yes. Perhaps not in more complicated cases.
That seems to disprove your “only correct policy” claim. I thought your “sequential equilibrium” line was trying to preempt this argument, but I can’t see how.
And I can’t see how to explain it without an example
While I wait, did you see anything in Aumann’s paper that hints at “the policy of planning-optimal-for-the first-move, then action-optimal-thereafter might do better”? Or is that your original research (to use Wikipedia-speak)? It occurs to me that if you’re correct about that, the authors of the paper should have realized it themselves and mentioned it somewhere, since it greatly strengthens their position.
Answering that is a bit tricky. If I am wrong, it is certainly “original research”. But my belief is based upon readings in game theory (including stuff by Aumann) which are not explicitly contained in that paper.
Please bear with me. I have a multi-player example in mind, but I hope to be able to find a single-player one which makes the reasoning clearer.
Regarding your last sentence, I must point out that the whole reason we are having this discussion is my claim to the effect that you don’t really understand their position, and hence cannot judge what does or does not strengthen it.
Ok, I now have at least a sketch of an example. I haven’t worked it out in detail, so I may be wrong, but here is what I think. In any scenario in which you gain and act on information after the planning stage, you should not use a recalculated planning-stage solution for any decisions after you have acted upon that information. Instead, you need to do the action-optimal analysis.
For example, let us complicate the absent-minded driver scenario that you diagrammed by adding an information-receipt and decision node prior to those two identical intersections. The driver comes in from the west and arrives at a T intersection where he can turn left(north) or right(south). At the intersection is a billboard advertising today’s lunch menu at Casa de Maria, his favorite restaurant. If the billboard promotes chile, he will want to turn right so as to have a good chance of reaching Maria’s for lunch. But if the billboard promotes enchiladas, which he dislikes, he probably wants to turn the other way and try for Marcello’s Pizza. Whether he turns right or left at the billboard, he will face two consecutive identical intersections (four identical intersections total). The day is cloudy, so he cannot tell whether he is traveling north or south.
Working this example in detail will take some work. Let me know if you think the work is necessary.
Once you have this, you can use it at the first intersection. But at the other intersections, you have some choices
It is a part of the problem statement that you can’t distinguish between being at any of the intersections. So you have to use the same algorithm at all of them.
either recalculate the planning-optimal solution each time
How are you getting this from their words? What about “this coordination can take place only before he starts out at the planning stage”? And “If he chose something else, or nothing at all, then at the action stage he will have some hard thinking to do”? Why would they say “hard thinking” if they meant “recalculate the planning-optimal solution”? (Especially when the planning-optimality calculation is simpler than the action-optimality calculation.)
In the comment section of Wei Dai’s post in question, taw and pengvado completed his solution so conclusively that if you really take the time to understand the object level (instead of the meta level where some people are apriori smarter because they won a prize), you can’t help but feel the snarking was justified :-)
1A. It may well be a wrong problem. if so it ought to be dissolved.
1B. If so, many theorists (including presumably nobel prize winners), have missed it since 1969.
1C. Your intuition should not be considered a persuasive argument, even by you.
2 . Even ignoring any singularitarian predictions, given the degree to which knowledge acceleration has already advanced, you should expect to see cases where old standards are blown away with seemingly little effort.
Maybe this isn’t one of those cases, but it should not surprise you if we learn that humanity as a whole has done more decision theory in the past few years than in all previous history.
Given that the similar accelerations are happening in many fields, there are probably several past-nobel-level advances by rank amateurs with no special genius.
OK, I’ve got some big guns pointed at me, so I need to respond. I need to respond intelligently and carefully. That will take some time. Within a week at most.
For a long time I also didn’t think that Newcomb’s Problem was worth thinking about. Then I read something by Eliezer that pointed out the connection to Prisoner’s Dilemma. (According to Prisoners’ Dilemma is a Newcomb Problem, others saw the connection as early as 1969.) See also my
Newcomb’s Problem vs. One-Shot Prisoner’s Dilemma where I explored how they are different as well.
I’m curious what you now think about my perspective on the Absent Minded Driver, on both the object level and meta level (assuming I convinced you that it wasn’t meant to be a snark). You’re the only person who has indicated actually having read Aumann et al.’s paper.
The possible connection between Newcomb and PD is seen by anyone who considers Jeffrey’s version of decision theory (EDT). So I have seen it mentioned by philosophers long before I had heard of EY. Game theorists, of course, reject this, unless they are analysing games with “free precommitment”. I instinctively reject it too, for what that is worth, though I am beginning to realize that publishing your unchangeable source code is pretty-much equivalent to free precommitment.
My analysis of your analysis of AMD is in my response to your comment below.
Seriously, Omega is not just counterfactual, he is impossible. Why do you guys keep asking us to believe so many impossible things before breakfast?
Omega is not obviously impossible: in theory, someone could scan your brain and simulate how you react in a specific situation. If you’re already an upload and running as pure code, this is even easier.
The question is particularly relevant when trying to develop a decision theory for artificial intelligences: there’s nothing impossible about the notion of two adversarial AIs having acquired each others’ source codes and basing their actions on how a simulated copy of the other would react. If you presume that this scenario is possible, and there seems to be no reason to assume that it wouldn’t be, then developing a decision theory capable of handling this situation is an important part of building an AI.
So far, the only effect that all the Omega-talk has had on me is to make me honestly suspect that you guys must be into some kind of mind-over-matter quantum woo.
What on Earth gives you that impression? I agree that scenarios with Omega wil have probably little impact on practical matters, at least in near future, but quantum woo?
In every case I have seen so far where Eliezer has denigrated the standard game solution because it fails to win, he has been analyzing a game involving a physically and philosophically impossible fictional situation.
Why is Omega physically impossible? What is philosophically impossible, in general?
So far, the only effect that all the Omega-talk has had on me is to make me honestly suspect that you guys must be into some kind of mind-over-matter quantum woo.
What on Earth gives you that impression?
Omega makes a decision to put the money in the box, or not. In my model of (MWI) reality, that results in a branching—there are now 2 worlds (one with money, one without). The only problem is, I don’t know which world I am in. Next, I decide whether to one-box or to two-box. In my model, that results in 4 possible worlds now. Or more precisely, someone who knows neither my decision nor Omega’s would count 4 worlds.
But now we are asked to consider some kind of weird quantum correlation between Omega’s choice and my own. Omega’s choice is an event within my own past light-cone. By the usual physical assumptions, my choice should not have any causal influence on his choice. But I am asked to believe that if I choose to two-box, then he will have chosen not to leave money, whereas if I just believe as Omega wishes me to believe, then my choice will make me rich by reaching back and altering the past (selecting my preferred history?). And you ask “What on Earth gives me the impression that this is quantum woo?”
Omega makes a decision to put the money in the box, or not. In my model of (MWI) reality, that results in a branching—there are now 2 worlds (one with money, one without). The only problem is, I don’t know which world I am in. Next, I decide whether to one-box or to two-box. In my model, that results in 4 possible worlds now. Or more precisely, someone who knows neither my decision nor Omega’s would count 4 worlds.
Incorrect. Omega’s decision is no more indeterministic than the output of a calculation. Asking (say) me “Does two plus two equal three?” does not create two worlds, one in which I say “yes” and one in which I say “no”—overwhelmingly I will tell you “no”.
As others have said. Omega-talk is possible in a purely classical world, and is clearer in a classical world. Omega simply scans my brain and deterministically decides whether to put the money in or not. Then I decide whether I take one or two of the boxes. To say my choice should not have any causal influence on his choice is misleading at least. It may be true (depending on how exactly one defines causality), however it doesn’t exclude correlations between the two choices simply because they are both consequences of a common cause (state of my brain and the relevant portion of the world immediately before the scenario begun).
There is no need to include quantumness or even MWI into this scenario, and no certain reason why quantum effects would prevent it from happening. That said, I don’t say that something similar is probably going to happen soon.
That’s the case if you somehow manage to use a quantum coin in your decision. Your decision could be close enough to deterministic that the measure of the words where you decide differently is billions of times or more smaller and can safely be neglected.
Because, in predicting my future decisions, he is performing Laplace demon computations based on Heisenberg demon measurements. And physics rules out such demons.
What is philosophically impossible, in general?
Anything which cannot consistently coexist with what is already known to exist
One possibility: Omega is running this universe as a simulation, and has already run a large number of earlier identical instances.
Ok, that is possible, I suppose. Though it does conflict, in a sense, with the claim that he put the money in the box before I made the decision whether to one-box or two-box. Because, in some sense, I already made that decision in all(?) of those earlier identical simulations.
It is far from sure that the decisions made by human brains rely heavily on quantum effects and that the relevant data can’t be obtained by some non-destructive scanning, without Heisenberg-demonic measurements. The Laplace-demon aspects is in fact a matter of precision. If Omega needed to simulate the brain precisely (unfortunately, the formulations of the paradox here on LW and in the subsequent discussions suggest this), then yes, Omega would have to be a demon. But the Newcomb’s paradox needn’t happen in its idealised version with 100% success of Omega’s predictions to be valid and interesting. If Omega is right only 87% of the time, the paradox still holds, and I don’t see any compelling reason why this should be impossible without postulating demonic abilities.
Have you read the original article? The payoff is less if you follow ordinary decision theory, and yet the whole point of decision theory is to maximize the payoff.
Yes, I read that article, and at least a half dozen articles along the same line, and dozens of pages of commentary. I also remember the first LW article that I read; something about making beliefs “pay rent” in anticipated experiences. Since I don’t anticipate receiving a visit from Omega or anyone else who can read my mind, please forgive me if I don’t take the superiority of revisionist decision theories seriously. Show me a credible example where it does better. One which doesn’t involve some kind of spooky transmission of information backward in time.
Since I don’t anticipate receiving a visit from Omega or anyone else who can read my mind, please forgive me if I don’t take the superiority of revisionist decision theories seriously.
You are missing the point. Newcomb’s problem, and other problems involving Omega, are unit tests for mathematical formalizations of decision theory. When a decision theory gets a contrived problem wrong, we don’t care because that scenario might appear in real life, but because it demonstrates that the math is wrong in a way that might make it subtly wrong on other problems, too.
I think you are missing the point. Newcomb’s problem is equivalent to dividing by zero. Decision theories aren’t supposed to behave well when abused in this way. If they behave badly on this problem, maybe it is the fault of the problem rather than the fault of the theory.
If someone can present a more robust decision theory, UDT or TDT, or whatever, which handles all the well formed problems just as well as standard game theory, and also handles the ill-formed problems like Newcomb in accord with EY’s intuitions, then I think that is great. I look forward to reading the papers and textbooks explaining that decision theory. But until they have gone through at least some serious process of peer review, please forgive me if I dismiss them as just so much woo and/or vaporware.
Incidentally, I specified “EY’s intuitions” rather than “correctness” as the criterion of success, because unless Omega actually appears and submits to a series of empirical tests, I can’t imagine a more respectable empirical criterion.
No, randomness is kind of a red herring. I shouldn’t have brought it up.
At one point I thought I had a kind of Dutch Book argument against Omega—if he could predict some future “random” event which I intended to use in conjunction with a mixed strategy, then I should be able to profit by making side bets “hedging” my choice with respect to Omega. But when I looked more carefully, it didn’t work.
I don’t understand. You are answering my “show me”? Standard game theory says to defect in both PD and TPD. You have a revisionist decision theory that does better?
TDT does better, yes. My apologies; I’d forgotten the manuscript hasn’t yet been released to the public. It should be soon, I think; it’s been in the review process for a while. If for some reason Eliezer changed his mind and decided not to publish it then I’d be somewhat surprised. I’m guessing he’s nervous because it’s his first opportunity to show academia that he’s a real researcher and not just a somewhat bright autodidact.
There was a decision theory workshop a few months ago and a bunch of decision theorists are still working on solving the comparably much harder problems that were introduced at that time. Decision theory is still unsolved but UDT/TDT/ADT/XDT are a lot closer to solving it than the ancient CDT/EDT/SDT.
On the assumption that SDT stands for Sequential Decision Theory, I would like to take a shot at explaining this one, as well as at clarifying the relationship among CDT, EDT, and SDT. Everyone feel free to amend and extend my remarks.
Start with simple Bayesian updating. This is a theory of knowing, not a theory of acting. It helps you to know about the world, but doesn’t tell you what to do with your knowlege (other than to get more knowlege). There are two ways you can go from here: SDT and EDT.
SDT is basically game theory as developed by Seldin and Harsanyi. It adds agents, actions, and preferences to the world of propositions which exists in simple Bayesianism. Given the preferences of each agent regarding the propositions, and the agents’ beliefs about the effects which their actions have on the truth or falshood of propositions regarding which they have preferences, SDT advises each agent on their choice of actions. It is “sequential” because the decisions have to be considered in strict temporal order. For example, in the Parfit’s hitchhiker problem, both the hitchhiker and the motorist probably wish that the hitchhiker decision to pay $100 could be made before the motorist decision whether to offer a ride. But, in SDT, the decisions can not be made in this reverse order. By the same token, you cannot observe the future before deciding in the present.
If at least some of the agents in SDT believe that some of the other agents are rational, then you have game theory and things can get complicated. On the other hand, if you have only one agent, or if none of the agents believe that the others are rational, then you have classical decision theory which goes back to Wald (1939).
EDT is a variant of single-agent SDT due to Richard Jeffrey (1960s). In it, actions are treated just like any other proposition, except that some agents can make decisions that set action-propositions to be either true or false. The most interesting thing about EDT is that it is relatively “timeless”. That is, if X is an action, and A is an agent, then (A does X) might be thought of as a proposition. Using ordinary propositional logic, you can build and reason with compound propositions such as P → (A does X), (A does X)&Q->P, or (A does X)->(B does Y). The “timeless” aspect to this is that “A does X” is interpreted as “Either A did X, or is currently doing X, or will do X; I don’t really care about when it happens”.
The thing that makes EDT into a decision theory is the rule which says roughly “Act so as to make your preferred propositions true. If EDT worked as well as SDT, it would definitely be considered better, if only because of Ockham’s razor. It is an extremely elegant and simple theory. And it does work remarkably well. The most famous case where it doesn’t work (at least according to the SDT fans) is Newcomb’s problem. SDT says to two-box (because your decision cannot affect Omega’s already frozen-in-time decision). EDT says to one box (because it can’t even notice that the causality goes the wrong way). SDT and EDT also disagree regarding the Hitchhiker.
CDT is an attempt to improve on both SDT and EDT. It seems to be a work in progress. There are two variants out there right now—one built by philosophers and the other primarily the work of economist Judea Pearl. (I think I prefer Pearl’s version.) CDT helps to clarify the relationship between causation and correlation in Bayesian epistemology (i.e. learning). It also clarifies the relationship between action-based propositions (which are modeled in both SDT and EDT as somehow getting their truth value from the free will of the agents, and other propositions which get their truth value from the laws of physics. In CDT (Pearl’s version, at least) an action can be both free and determined—the flexibility reminds me of the compatibilist dissolution of the free will question which is suggested by the LW sequences.
I don’t know whether that summary answers the question you wanted answered, but I’m pretty sure the corrections I am likely to receive will answer the questions I want answered. :)
I think ADT is only described on Vladimir Nesov’s blog (if there) and XDT nowhere findable. ADT stands for Ambient Decision Theory. Unfortunately there’s no comprehensive and easy summary of any of the modern decision theories anywhere. Hopefully Eliezer publishes his TDT manuscript soon.
I coined the name XDT here. I think Anna Salamon and Steve Rayhawk had come up with essentially the same idea prior to that (and have explored its implications more deeply, but not in published form).
Thanks. I couldn’t find any references to ADT on Vladimir Nesov’s blog but I only had a quick scan so maybe I missed it, will have a better look later. And I can now remember that series of comments on XDT but my mind didn’t connect to it, thanks for the link.
TDT does better, yes. My apologies; I’d forgotten the manuscript hasn’t yet been released to the public. It should be soon, I think; it’s been in the review process for a while.
Wow. I didn’t realise Eliezer had decided to actually release something formally. My recollection was that he was refusing to work on it unless someone promised him a PhD.
Does better how? By cooperating? By achieving an reverse-Omega-like stance and somehow constraining the other player to cooperate, conditionally on cooperating ourselves? I am completely mystified. I guess I will have to wait for the paper(s).
As I said, I think your correspondents are in rather a muddle—and are discussing a completely different and rather esoteric PD case—where the agents can see and verify each other’s source code.
Thanks for the link. It was definitely telegraphic, but I think I got a pretty good notion where he is coming from with this, and also a bit about where he is going. I’m sure you remember the old days back at sci.bio.evolution talking about the various complications with the gene-level view of selection and Hamilton’s rule. Well, give another read to EY’s einsatz explanation of TDT:
The one-sentence version is: Choose as though controlling the logical output of the abstract computation you implement, including the output of all other instantiations and simulations of that computation.
Does that remind you of anything? “As you are deciding how the expression of you as a gene is going to affect the organism, remember to take into account that you are deciding for all of the members of your gene clone, and that changing the expression of your clone in other organisms is going to have an impact on the fitness of your own containing organism.” Now that is really cool. For the first time I begin to see how different decision theories might be appropriate for different meanings of the term “rational agent”.
I can’t claim to have understood everything EY wrote in that sketch, but I did imagine that I understood his concerns regarding “contrafactual surgery”. I want to get a hold of a preprint of the paper, when it is ready.
I think your correspondents are in rather a muddle—and are discussing a completely different and rather esoteric PD case—where the agents can see and verify each other’s source code. In which case, C-C is perfectly possible.
Since I don’t anticipate receiving a visit from Omega or anyone else who can read my mind, please forgive me if I don’t take the superiority of revisionist decision theories seriously. Show me a credible example where it does better.
You have been given at least one such example in this thread and even had you not the process of taking an idealised problem and creating more mundane example should be one you are familiar with if you are as well versed in the literature as you claim.
Where was I given such an example? The only example I saw was of an unreliable Omega, an Omega who only gets it right 90% of the time.
If that is the example you mean, then (1) I agree that it adds unnecessary complexity by bringing in irrelevant considerations, and (2) I claim it is still f’ing impossible.
Impossible things need to have zero-probability priors.
0 and 1 are not probabilities. I certainly don’t have a prior of 0 that Omega’s existence is impossible; he’s not defined in a contradictory fashion, and even if he was I harbor the tiniest bit of doubt that I’m wrong about how contradictions work.
If you were a Bayesian and assigned 0 probability to 2+2=5, you’d be in unrecoverable epistemic trouble if you turned out to be wrong about that. See How to convince me 2+2=3.
EY to the contrary, I remain smug in my evaluation p(2+2=5)=0. Of all the evidences that Eliezer offered, the only one to convince me was the one which demonstrated to me that I was confused about the meaning of the digit 5. Yes, by Cromwell’s rule, I think it possible I might be mistaken about how to count. “1, 2, 3, 5, 6, 4, 7”, I recite to myself.
“Yes, I had been wrong about that. Thanks for correcting me.”
I might then write down p(Eliezer Yupkowski is the guru of Less Wrong)=0.999999999. Once again, I would be mistaken. It is “Yudkowski”, not “Yupkowski”) But in neither case am I in unrecoverable epistemic trouble. Those were typos. Correcting them is a simple search-and-replace, not a Bayesian updating. Or so I understand.
I might then write down p(Eliezer Yupkowski is the guru of Less Wrong)=0.999999999. Once again, I would be mistaken. It is “Yudkowski”, not “Yupkowski”) But in neither case am I in unrecoverable epistemic trouble. Those were typos. Correcting them is a simple search-and-replace, not a Bayesian updating. Or so I understand.
It’s Yudkowsky. Might want to update your general confidence evaluations.
I might then write down p(Eliezer Yupkowski is the guru of Less Wrong)=0.999999999. Once again, I would be mistaken. It is “Yudkowski”, not “Yupkowski”
Ah! So I need to assign priors to three hypotheses. (1) Omega is a magician (i.e. illusion artist) (2) Omega had bribed people to lie about his past success. (3) He is what he claims.
So I assign a prior of zero probability to hypothesis #3, and cheerfully one-box using everyday decision theory.
You don’t seem to be entering into the spirit of the problem. You are “supposed” to reach the conclusion that there’s a good chance that Omega can predict your actions in this domain pretty well—from what he knows about you—after reading the premise of the problem.
If you think that’s not a practical possibility, then I recommend that you imagine yourself as a deterministic robot—where such a scenario becomes more believable—and then try the problem again.
If I imagine myself as a deterministic robot, who knows that he is a deterministic robot, I am no longer able to maintain the illusion that I care about this problem.
I like this version! Now the answer seems quite obvious.
In this case, I would design the robot to be a one-boxer. And I would harbour the secret hope that a stray cosmic ray will cause the robot to pick both boxes anyway.
Let me ask the question this way: What evidence do you have that the standard solution to the one-shot PD can be improved upon without creating losses elsewhere? My impression is that you are being driven by wishful thinking and misguided intuition.
For what it’s worth, I have written programs that cooperate on the prisoner’s dilemma if and only if their opponent will cooperate, without caring about the opponent’s rituals of cognition, only about his behaviour.
Unfortunately, this margin is too small to contain them, I mean, they’re not ready for prime time. I’ll probably write up a post on that in the near future.
Yes, back when we discussed Newcomblike problems frequently I more or less used a form letter to reply to that objection. Any useful treatment of Newcomblike problems will specify explicitly or implicitly how Omega will handle (quantum) randomness if it is allowed. The obvious response for Omega is to either give you nothing (or maybe a grenade!) for being a smart ass or, more elegantly, handle the reward given in commensurate manner to the probabilities. If probabilistic decisions are to be allowed then an Omega that can handle probabilistic decisions quite clearly needs to be supplied.
I downvoted the parent. How on earth is Perplexed comparing LW conventional wisdom to that of Nobel prize winning economists when he thinks coin tossing is a big deal?
At the risk of appearing stupid, I have to ask: exactly what is a “useful treatment of Newcomb-like problems” used for?
So far, the only effect that all the Omega-talk has had on me is to make me honestly suspect that you guys must be into some kind of mind-over-matter quantum woo.
Seriously, Omega is not just counterfactual, he is impossible. Why do you guys keep asking us to believe so many impossible things before breakfast? Jaynes says not to include impossible propositions among the conditions in a conditional probability. Bad things happen if you do. Impossible things need to have zero-probability priors. Omega just has no business hanging around with honest Bayesians.
When I read that you all are searching for improved decision theories that “solve” the one-shot prisoner’s dilemma and the one-shot Parfit hitchhiker, I just cringe. Surely you shouldn’t change the standard, well-established, and correct decision theories. If you don’t like the standard solutions, you should instead revise the problems from unrealistic one-shots to more realistic repeated games or perhaps even more realistic games with observers—observers who may play games with you in the future.
In every case I have seen so far where Eliezer has denigrated the standard game solution because it fails to win, he has been analyzing a game involving a physically and philosophically impossible fictional situation.
Let me ask the question this way: What evidence do you have that the standard solution to the one-shot PD can be improved upon without creating losses elsewhere? My impression is that you are being driven by wishful thinking and misguided intuition.
Here’s another way of looking at the situation that may or may not be helpful. Suppose I ask you, right here and now, what you’d do in the hypothetical future Parfit’s Hitchhiker scenario if your opponent was a regular human with Internet access. You have several options:
Answer truthfully that you’d pay $100, thus proving that you don’t subscribe to CDT or EDT. (This is the alternative I would choose.)
Answer that you’d refuse to pay. Now you’ve created evidence on the Internet, and if/when you face the scenario in real life, the driver will Google your name, check the comments on LW and leave you in the desert to die. (Assume the least convenient possible world where you can’t change or delete your answer once it’s posted.)
Answer that you’d pay up, but secretly plan to refuse. This means you’d be lying to us here in the comments—surely not a very nice thing to do. But if you subscribe to CDT with respect to utterances as well as actions, this is the alternative you’re forced to choose. (Which may or may not make you uneasy about CDT.)
What makes me uneasy is the assumption I wouldn’t want to pay $100 to somebody who rescued me from the desert. Given that, lying to people whom I don’t really know should be a piece of cake!
I would of course choose option #1, adding that, due to an affliction giving me a trembling hand, I tend to get stranded in the desert and the like a lot and hence that I would appreciate it if he would spread the story of my honesty among other drivers. I might also promise to keep secret the fact of his own credulity in this case, should he ask me to. :)
I understand quite well that the best and simplest way to appear honest is to actually be honest. And also that, as a practical matter, you never really know who might observe your selfish actions and how that might hurt you in the future. But these prudential considerations can already be incorporated into received decision theory (which, incidentally, I don’t think matches up with either CDT or EDT—at least as those acronyms seem to be understood here.) We don’t seem to need TDT and UDT to somehow glue them in to the foundations.
Hmmm. Is EY perhaps worried that an AI might need need even stronger inducements toward honesty? Maybe it would, but I don’t see how you solve the problem by endowing the AI with a flawed decision theory.
...What?
Also, it doesn’t matter if he’s impossible. He’s an easy way to tack on arbitrary rules to hypotheticals without overly tortured explanations, because people are used to getting arbitrary rules from powerful agents.
It’s also impossible for a perfectly Absent Minded Driver to come to one of only two possible intersections with 3 destinations with known payoffs and no other choices. To say nothing of the impossibly horrible safety practices of our nation’s hypothetical train system.
Are you sure? I’m not objecting to the arbitrary payoffs or complaining because he doesn’t seem to be maximizing his own utility. I’m objecting to his ability to predict my actions. Give me a scenario which doesn’t require me to assign a non-zero prior to woo and in which a revisionist decision theory wins. If you can’t, then your “improved” decision theory is no better than woo itself.
Regarding the Absent Minded Driver, I didn’t recognize the reference. Googling, I find a .pdf by one of my guys (Nobelist Robert Aumann) and an LW article by Wei-Dai. Cool, but since it is already way past my bedtime, I will have to read them in the morning and get back to you.
The only ‘woo’ here seems to be your belief that your actions are not predictable (even in principle!). Even I can predict your actions within some tolerances, and we do not need to posit that I am a superintelligence! Examples: you will not hang yourself to death within the next five minutes, and you will ever make another comment on Less Wrong.
“ever”? No, “never”.
Wha?
In case it wasn’t clear, it was a one-off prediction and I was already correct.
In case mine wasn’t clear, it was a bad Gilbert & Sullivan joke. Deservedly downvoted. Apparently.
You need a little more context/priming or to make the joke longer for anyone to catch this. Or you need to embed it in a more substantive and sensible reply. Otherwise it will hardly ever work.
Counterexample
I’d call that a long joke, wouldn’t you?
See what I mean? I made it long and it still didn’t work. :)
I wasn’t sure, so I held off posting my reply (a decision I now regret). It would have been, “Well, hardly ever.”
Why? What about you is fundamentally logically impossible to predict?
Do you not find that you often predict the actions of others? (ie. giving them gifts that you know they’ll like) And that others predict your reactions? (ie. choosing not to give you spider-themed horror movies if you’re arachnophobic)
Omega is a perfect super-intelligence, existing in a computer simulation like universe that can be modeled by a set of physical laws and a very long string of random numbers. Omega knows the laws and the numbers.
Ok, I’ve read the paper(most of it) and Wei-Dai’s article now. Two points.
In a sense, I understand how you might think that the Absent Minded Driver is no less contrived and unrealistic than Newcomb’s Paradox. Maybe different people have different intuitions as to what toy examples are informative and which are misleading. Someone else (on this thread?) responded to me recently with the example of frictionless pulleys and the like from physics. All I can tell you is that my intuition tells me that the AMD, the PD, frictionless pulleys,and even Parfit’s Hitchhiker all strike me as admirable teaching tools, whereas Newcomb problems and the old questions of irrestable force vs immovable object in physics are simply wrong problems which can only create confusion.
Reading Wei-Dai’s snarking about how the LW approach to decision theory (with zero published papers to date) is so superior to the confusion in which mere misguided Nobel laureates struggle—well, I almost threw up. It is extremely doubtful that I will continue posting here for long.
It wasn’t meant to be a snark. I was genuinely trying to figure out how the “LW approach” might be superior, because otherwise the most likely explanation is that we’re all deluded in thinking that we’re making progress. I’d be happy to take any suggestions on how I could have reworded my post so that it sounded less like a snark.
Wei-Dai wrote a post entitled The Absent-Minded Driver which I labeled “snarky”. Moreover, I suggested that the snarkiness was so bad as to be nauseating, so as to drive reasonable people to flee in horror from LW and SAIA. I here attempt to defend these rather startling opinions. Here is what Wei-Dai wrote that offended me:
The paper that Wei-Dai reviews is “The Absent-Minded Driver” by Robert J. Aumann, Sergiu Hart, and Motty Perry. Wei-Dai points out, rather condescendingly:
Wei-Dai then proceeds to give a competent description of the problem and the standard “planning-optimality” solution of the problem. Next comes a description of an alternative seductive-but-wrong solution by Piccione and Rubinstein. I should point that everyone—P&R, Aumann, Hart, and Perry, Wei-Dai, me, and hopefully you who look into this—realizes that the alternative P&R solution is wrong. It gets the wrong result. It doesn’t win. The only problem is explaining exactly where the analysis leading to that solution went astray, and in explaining how it might be modified so as to go right. Making this analysis was, as I see it, the whole point of both papers—P&R and Aumann et al. Wei-Dai describes some characteristics of Aumann et al’s corrected version of the alternate solution. Then he (?) goes horribly astray:
But, as anyone who reads the paper carefully should see, they weren’t arguing for action-optimality as the solution. They never abandoned planning optimality. Their point is that if you insist on reasoning in this way, (and Seldin’s notion of “subgame perfection” suggests some reasons why you might!) then the algorithm they call “action-optimality” is the way to go about it.
But Wei-Dai doesn’t get this. Instead we get this analysis of how these brilliant people just haven’t had the educational advantages that LW folks have:
Let me just point out that the reason it is true that “they never argued against it” is that they had already argued for it. Check out the implications of their footnote #4!
Ok, those are the facts, as I see them. Was Wei-Dai snarky? I suppose it depends on how you define snarkiness. Taboo “snarky”. I think that he was overbearingly condescending without the slightest real reason for thinking himself superior. “Snarky” may not be the best one-word encapsulation of that attitude, but it is the one I chose. I am unapologetic. Wei-Dai somehow came to believe himself better able to see the truth than a Nobel laureate in the Nobel laureate’s field. It is a mistake he would not have made had he simply read a textbook or taken a one-semester course in the field. But I’m coming to see it as a mistake made frequently by SIAI insiders.
Let me point out that the problem of forgetful agents may seem artificial, but it is actually extremely important. An agent with perfect recall playing the iterated PD, knowing that it is to be repeated exactly 100 times, should rationally choose to defect. On the other hand, if he cannot remember how many iterations remain to be played, and knows that the other player cannot remember either, should cooperate by playing Tit-for-Tat or something similar.
Well, that is my considered response on “snarkiness”. I still have to respond on some other points, and I suspect that, upon consideration, I am going to have to eat some crow. But I’m not backing down on this narrow point. Wei-Dai blew it in interpreting Aumann’s paper. (And also, other people who know some game theory should read the paper and savor the implications of footnote #4. It is totally cool).
How is Wei Dai being condescending there? He’s pointing out how weak it is to dismiss people with these credentials by just calling them crazy. ETA: In other words, it’s an admonishment directed at LWers.
That, at any rate, was my read.
I’m sure it would be Wei-Dai’s read as well. The thing is, if Wei-Dai had not mistakenly come to the conclusion that the authors are wrong and not as enlightened as LWers, that admonishment would not be necessary. I’m not saying he condescends to LWers. I say he condescends to the rest of the world, particularly game theorists.
Are you essentially saying you are nauseated because Wei Dai disagreed with the authors?
No. Not at all. It is because he disagreed through the wrong channels, and then proceeded to propose rather insulting hypotheses as to why they had gotten it wrong.
Just read that list of possible reasons! And there are people here arguing that “of course we want to analyze the cause of mistakes”. Sheesh. No wonder folks here are so in love with Evolutionary Psychology.
Ok, I’m probably going to get downvoted to hell because of that last paragraph. And, you know what, that downvoting impulse due to that paragraph pretty much makes my case for why Wei Dai was wrong to do what he did. Think about it.
Interestingly enough I think that it is this paragraph that people will downvote, and not the one above. Mind you, the premise in “No wonder folks here are so in love with Evolutionary Psychology.” does seem so incredibly backward that I almost laughed.
I can understand your explanation here. Without agreeing with it myself I can see how it follows from your premises.
I’m having trouble following you.
Are you saying that you read him differently, and that he would somehow be misinterpreting himself?
The admonishment is necessary if LWers are likely to wrongly dismiss Aumann et al. as “crazy”. In other words, to think that the admonishment is necessary is to think that LWers are too inclined to dismiss other people as crazy
I got that. Who said anything about condescending to LWers?
Huh?? Surely, you troll. I am saying that Wei-Dai’s read would likely be the same as yours: that he was not condescending; that he was in fact cautioning his readers against looking down on the poor misguided Nobelists who, after all, probably had good reasons for being so mistaken. There, but for the grace of EY, go we.
Or was I really that unclear?
Condescension is a combination of content and context. When you isolated that quote as especially condescending, I thought that you read something within it that was condescending. I was confused, because the quote could just as well have come from a post arguing that LWers ought to believe that Aumann et al. are right.
It now looks like you and I read the intrinsic meaning of the quote in the same way. The question then is, does that quote, placed in context, somehow make the overall post more condescending than it already was? Wei had already said that his treatment of the AMD was better than that of Aumann et al.. He had already said that these prestigious researchers got it wrong. Do you agree that if this were true, if the experts got it wrong, then we ought to try to understand how that happened, and not just dismiss them as crazy?
Whatever condescension occurred, it occurred as soon as Wei said that he was right and Aumann et al. were wrong. How can drawing a rational inference from that belief make it more condescending?
In this light I can see where ‘condescension’ fits in. There is a difference between ‘descending to be with’ and just plain ‘being way above’. For example we could label “they are wrong” as arrogant, “they are wrong but we can empathise with them and understand their mistake” as condescending and “They are wrong, that’s the kind of person Nobel prizes go to these days?” as “contemptuous”—even though they all operate from the same “I consider myself above in this instance” premise. Wei’s paragraph could then be considered to be transferring weight from arrogance and contempt into condescension.
(I still disapprove of Perplexed’s implied criticism.)
Okay, I can see this distinction. I can see how, as a matter of social convention, “they are wrong but we should understand their mistake” could come across as more condescending than just “they are wrong”. But I really don’t like that convention. If an expert is wrong, we really do have an obligation to understand how that happened. Accepting that obligation shouldn’t be stigmatized as condescending. (Not that you implied otherwise.)
“They are probably not crazy” strikes me as “damning with faint praise”. IMHO, it definitely raises the overall condescension level.
No. Peons claim lords are wrong all the time. It is not even impolite, if you are willing to admit your mistake and withdraw your claim reasonably quickly.
Condescension starts when you attempt to “charitably” analyze the source of the error.
Of course. But if I merely had good reason to believe they were wrong, then my most urgent next step would be to determine whether it were true that they got it wrong. I would begin by communicating with the experts, either privately or through the peer-reviewed literature, so as to get some feedback as to whether they were wrong or I was mistaken. If it does indeed turn out that they were wrong, I would let them take the first shot at explaining the causes of their mistake. I doubt that I would try to analyze the cause of the mistake myself unless I were a trained historian dealing with a mistake at least 50 years old. Or, if I did try (and I probably have), I would hope that someone would point out my presumption.
Preliminary notes: You can call me “Wei Dai” (that’s firstname lastname). “He” is ok. I have taken a graduate level course in game theory (where I got a 4.0 grade, in case you suspect that I coasted through it), and have Fudenberg and Tirole’s “Game Theory” and Joyce’s “Foundations of Causal Decision Theory” as two of the few physical books that I own.
I can’t see where they made this point. At the top of Section 4, they say “How, then, should the driver reason at the action stage?” and go on directly to describe action-optimality. If they said something like “One possibility is to just recompute and apply the planning-optimal solution. But if you insist …” please point out where. See also page 108:
If Aumann et al. endorse using planning-optimality at the action stage, why would they say the driver has some hard thinking to do? Again, why not just recompute and apply the planning-optimal solution?
I also do not see how subgame perfection is relevant here. Can you explain?
This footnote?
Since p* is the action-optimal solution, they are pointing out the formal relationship between their notion of action-optimality and Nash equilibrium. How is this footnote an argument for “it” (it being “recomputing the planning-optimal decision at each intersection and carrying it out”)?
Ok, so it is me who is convicted of condescending without having the background to justify it. :( FWIW I have never taken a course, though I have been reading in the subject for more than 45 years.
My apologies. More to come on the substance.
Relevance of Subgame perfection. Seldin suggested subgame perfection as a refinement of Nash equilibrium which requires that decisions that seemed rational at the planning stage ought to still seem rational at the action stage. This at least suggests that we might want to consider requiring “subgame perfection” even if we only have a single player making two successive decisions.
Relevance of Footnote #4. This points out that one way to think of problems where a single player makes a series of decisions is to pretend that the problem has a series of players making the decisions—one decision per player, but that these fictitious players are linked in that they all share the same payoffs (but not necessarily the same information). This is a standard “trick” in game theory, but the footnote points out that in this case, since both fictitious players have the same information (because of the absent-mindedness) the game between driver-version-1 and driver-version-2 is symmetric, and that is equivalent to the constraint p1 = p2.
Does Footnote #4 really amount to “they had already argued for [just recalculating the planning-optimal solution]”? Well, no it doesn’t really. I blew it in offering that as evidence. (Still think it is cool, though!)
Do they “argue for it” anywhere else? Yes, they do. Section 5, where they apply their methods to a slightly more complicated example, is an extended argument for the superiority of the planning-optimal solution to the action-optimal solutions. As they explain, there can be multiple action-optimal solutions, even if there is only one (correct) planning-optimal solution, and some of those action-optimal solutions are wrong *even though they appear to promise a higher expected payoff than does the planning optimal solution.
I really don’t see why you are having so much trouble parsing this. “If indeed he chose p1 , there is no problem” is an endorsement of the correctness of the planning-optimal solution. The sentence dealing with p2 and p3 asserts that, if you mistakenly used p2 for your first decision, then you best follow-up is to remain consistent and use p2 for your remaining two choices. The paragraph you quote to make your case is one I might well choose myself to make my case.
Edit: There are some asterisks in variable names in the original paper which I was unable to make work with the italics rules on this site. So “p2” above should be read as “p 2″
It is a statement that the planning-optimal action is the correct one, but it’s not an endorsement that it is correct to use the planning-optimality algorithm to compute what to do when you are already at an intersection. Do you see the difference?
ETA (edited to add): According to my reading of that paragraph, what they actually endorse is to compute the planning-optimal action at START, remember that, then at each intersection, compute the set of action-optimal actions, and pick the element of the set that coincides with the planning-optimal action.
BTW, you can use “\” to escape special characters like “*” and “_”.
Thx for the escape character info. That really ought to be added to the editing help popup.
Yes, I see the difference. I claim that what they are saying here is that you need to do the planning-optimal calculation in order to find p*1 as the unique best solution (among the three solutions that the action-optimal method provides). Once you have this, you can use it at the first intersection. But at the other intersections, you have some choices: either recalculate the planning-optimal solution each time, or write down enough information so that you can recognize that p*1 is the solution you are already committed to among the three (in section 5) solutions returned by the action-optimality calculation.
ETA in response to your ETA. Yes they do. Good point. I’m pretty sure there are cases more complicated than this perfectly amnesiac driver where that would be the only correct policy. (ETA:To be more specific, cases where the planning-optimal solution is not a sequential equilibrium). But then I have no reason to think that UDT would yield the correct answer in those more complicated cases either.
I deleted my previous reply since it seems unnecessary given your ETA.
What would be the only correct policy? What I wrote after “According to my reading of that paragraph”? If so, I don’t understand your “cases where the planning-optimal solution is not a sequential equilibrium”. Please explain.
Yes.
I would have thought it would be self explanatory.
It looks like I will need to construct and analyze examples slightly more complicated that the Absent Minded Driver. That may take a while. Questions before I start: Does UDT encompass game theory, or is it limited to analyzing single-player situations? Is UDT completely explained in your postings, or is it, like TDT, still in the process of being written up?
Wei has described a couple versions of UDT. His descriptions seemed to me to be mathematically rigorous. Based on Wei’s posts, I wrote this pdf, which gives just the definition of a UDT agent (as I understand it), without motivation or justification.
The difficulty with multiple agents looks like it will be very hard to get around within the UDT framework. UDT works essentially by passing the buck to an agent who is at the planning stage*. That planning-stage agent then performs a conventional expected-utility calculation.
But some scenarios seem best described by saying that there are multiple planning-stage agents. That means that UDT is subject to all of the usual difficulties that arise when you try to use expected utility alone in multiplayer games (e.g., prisoners dilemma). It’s just that these difficulties arise at the planning stage instead of at the action stage directly.
*Somewhat more accurately, the buck is passed to the UDT agent’s simulation of an agent who is at the planning stage.
What I meant was, what point were you trying to make with that statement? According to Aumann’s paper, every planning-optimal solution is also an action-optimal solution, so the decision procedure they endorse will end up picking the planning-optimal solution. (My complaint is just that it goes about it in an unnecessarily round-about way.) If theirs is a correct policy, then the policy of just recomputing the planning-optimal solution must also be correct. That seems to disprove your “only correct policy” claim. I thought your “sequential equilibrium” line was trying to preempt this argument, but I can’t see how.
Pretty much single-player for now. A number of people are trying to extend the ideas to multi-player situations, but it looks really hard.
No, it’s not being written up further. (Nesov is writing up some of his ideas, which are meant to be an advance over UDT.)
My understanding of their paper has changed somewhat since we began this discussion. I now believe that repeating the planning-optimal analysis at every decision node is only guaranteed to give ideal results in simple cases like this one in which every decision point is in the same information set. In more complicated cases, I can imagine that the policy of planning-optimal-for-the first-move, then action-optimal-thereafter might do better. I would need to construct an example to assert this with confidence.
In this simple example, yes. Perhaps not in more complicated cases.
And I can’t see how to explain it without an example
While I wait, did you see anything in Aumann’s paper that hints at “the policy of planning-optimal-for-the first-move, then action-optimal-thereafter might do better”? Or is that your original research (to use Wikipedia-speak)? It occurs to me that if you’re correct about that, the authors of the paper should have realized it themselves and mentioned it somewhere, since it greatly strengthens their position.
Answering that is a bit tricky. If I am wrong, it is certainly “original research”. But my belief is based upon readings in game theory (including stuff by Aumann) which are not explicitly contained in that paper.
Please bear with me. I have a multi-player example in mind, but I hope to be able to find a single-player one which makes the reasoning clearer.
Regarding your last sentence, I must point out that the whole reason we are having this discussion is my claim to the effect that you don’t really understand their position, and hence cannot judge what does or does not strengthen it.
Ok, I now have at least a sketch of an example. I haven’t worked it out in detail, so I may be wrong, but here is what I think. In any scenario in which you gain and act on information after the planning stage, you should not use a recalculated planning-stage solution for any decisions after you have acted upon that information. Instead, you need to do the action-optimal analysis.
For example, let us complicate the absent-minded driver scenario that you diagrammed by adding an information-receipt and decision node prior to those two identical intersections. The driver comes in from the west and arrives at a T intersection where he can turn left(north) or right(south). At the intersection is a billboard advertising today’s lunch menu at Casa de Maria, his favorite restaurant. If the billboard promotes chile, he will want to turn right so as to have a good chance of reaching Maria’s for lunch. But if the billboard promotes enchiladas, which he dislikes, he probably wants to turn the other way and try for Marcello’s Pizza. Whether he turns right or left at the billboard, he will face two consecutive identical intersections (four identical intersections total). The day is cloudy, so he cannot tell whether he is traveling north or south.
Working this example in detail will take some work. Let me know if you think the work is necessary.
Ok, I see. I’ll await your example.
It is a part of the problem statement that you can’t distinguish between being at any of the intersections. So you have to use the same algorithm at all of them.
How are you getting this from their words? What about “this coordination can take place only before he starts out at the planning stage”? And “If he chose something else, or nothing at all, then at the action stage he will have some hard thinking to do”? Why would they say “hard thinking” if they meant “recalculate the planning-optimal solution”? (Especially when the planning-optimality calculation is simpler than the action-optimality calculation.)
You can use a backslash to escape special characters in markdown.
If you type \*, that will show up as * in the posted text.
In the comment section of Wei Dai’s post in question, taw and pengvado completed his solution so conclusively that if you really take the time to understand the object level (instead of the meta level where some people are apriori smarter because they won a prize), you can’t help but feel the snarking was justified :-)
1A. It may well be a wrong problem. if so it ought to be dissolved.
1B. If so, many theorists (including presumably nobel prize winners), have missed it since 1969.
1C. Your intuition should not be considered a persuasive argument, even by you.
2 . Even ignoring any singularitarian predictions, given the degree to which knowledge acceleration has already advanced, you should expect to see cases where old standards are blown away with seemingly little effort.
Maybe this isn’t one of those cases, but it should not surprise you if we learn that humanity as a whole has done more decision theory in the past few years than in all previous history.
Given that the similar accelerations are happening in many fields, there are probably several past-nobel-level advances by rank amateurs with no special genius.
OK, I’ve got some big guns pointed at me, so I need to respond. I need to respond intelligently and carefully. That will take some time. Within a week at most.
A couple more comments:
For a long time I also didn’t think that Newcomb’s Problem was worth thinking about. Then I read something by Eliezer that pointed out the connection to Prisoner’s Dilemma. (According to Prisoners’ Dilemma is a Newcomb Problem, others saw the connection as early as 1969.) See also my Newcomb’s Problem vs. One-Shot Prisoner’s Dilemma where I explored how they are different as well.
I’m curious what you now think about my perspective on the Absent Minded Driver, on both the object level and meta level (assuming I convinced you that it wasn’t meant to be a snark). You’re the only person who has indicated actually having read Aumann et al.’s paper.
The possible connection between Newcomb and PD is seen by anyone who considers Jeffrey’s version of decision theory (EDT). So I have seen it mentioned by philosophers long before I had heard of EY. Game theorists, of course, reject this, unless they are analysing games with “free precommitment”. I instinctively reject it too, for what that is worth, though I am beginning to realize that publishing your unchangeable source code is pretty-much equivalent to free precommitment.
My analysis of your analysis of AMD is in my response to your comment below.
Omega is not obviously impossible: in theory, someone could scan your brain and simulate how you react in a specific situation. If you’re already an upload and running as pure code, this is even easier.
The question is particularly relevant when trying to develop a decision theory for artificial intelligences: there’s nothing impossible about the notion of two adversarial AIs having acquired each others’ source codes and basing their actions on how a simulated copy of the other would react. If you presume that this scenario is possible, and there seems to be no reason to assume that it wouldn’t be, then developing a decision theory capable of handling this situation is an important part of building an AI.
What on Earth gives you that impression? I agree that scenarios with Omega wil have probably little impact on practical matters, at least in near future, but quantum woo?
Why is Omega physically impossible? What is philosophically impossible, in general?
Omega makes a decision to put the money in the box, or not. In my model of (MWI) reality, that results in a branching—there are now 2 worlds (one with money, one without). The only problem is, I don’t know which world I am in. Next, I decide whether to one-box or to two-box. In my model, that results in 4 possible worlds now. Or more precisely, someone who knows neither my decision nor Omega’s would count 4 worlds.
But now we are asked to consider some kind of weird quantum correlation between Omega’s choice and my own. Omega’s choice is an event within my own past light-cone. By the usual physical assumptions, my choice should not have any causal influence on his choice. But I am asked to believe that if I choose to two-box, then he will have chosen not to leave money, whereas if I just believe as Omega wishes me to believe, then my choice will make me rich by reaching back and altering the past (selecting my preferred history?). And you ask “What on Earth gives me the impression that this is quantum woo?”
Incorrect. Omega’s decision is no more indeterministic than the output of a calculation. Asking (say) me “Does two plus two equal three?” does not create two worlds, one in which I say “yes” and one in which I say “no”—overwhelmingly I will tell you “no”.
Your model ought to be branching at every subatomic event, not at every conscious intelligent choice.
This makes reality (even humans) predictable.
As others have said. Omega-talk is possible in a purely classical world, and is clearer in a classical world. Omega simply scans my brain and deterministically decides whether to put the money in or not. Then I decide whether I take one or two of the boxes. To say my choice should not have any causal influence on his choice is misleading at least. It may be true (depending on how exactly one defines causality), however it doesn’t exclude correlations between the two choices simply because they are both consequences of a common cause (state of my brain and the relevant portion of the world immediately before the scenario begun).
There is no need to include quantumness or even MWI into this scenario, and no certain reason why quantum effects would prevent it from happening. That said, I don’t say that something similar is probably going to happen soon.
That’s the case if you somehow manage to use a quantum coin in your decision. Your decision could be close enough to deterministic that the measure of the words where you decide differently is billions of times or more smaller and can safely be neglected.
Because, in predicting my future decisions, he is performing Laplace demon computations based on Heisenberg demon measurements. And physics rules out such demons.
Anything which cannot consistently coexist with what is already known to exist
One possibility: Omega is running this universe as a simulation, and has already run a large number of earlier identical instances.
There may be many less obvious possibilities, even if you require Omega to be certain rather than just very sure.
Ok, that is possible, I suppose. Though it does conflict, in a sense, with the claim that he put the money in the box before I made the decision whether to one-box or two-box. Because, in some sense, I already made that decision in all(?) of those earlier identical simulations.
It is far from sure that the decisions made by human brains rely heavily on quantum effects and that the relevant data can’t be obtained by some non-destructive scanning, without Heisenberg-demonic measurements. The Laplace-demon aspects is in fact a matter of precision. If Omega needed to simulate the brain precisely (unfortunately, the formulations of the paradox here on LW and in the subsequent discussions suggest this), then yes, Omega would have to be a demon. But the Newcomb’s paradox needn’t happen in its idealised version with 100% success of Omega’s predictions to be valid and interesting. If Omega is right only 87% of the time, the paradox still holds, and I don’t see any compelling reason why this should be impossible without postulating demonic abilities.
Have you read the original article? The payoff is less if you follow ordinary decision theory, and yet the whole point of decision theory is to maximize the payoff.
Yes, I read that article, and at least a half dozen articles along the same line, and dozens of pages of commentary. I also remember the first LW article that I read; something about making beliefs “pay rent” in anticipated experiences. Since I don’t anticipate receiving a visit from Omega or anyone else who can read my mind, please forgive me if I don’t take the superiority of revisionist decision theories seriously. Show me a credible example where it does better. One which doesn’t involve some kind of spooky transmission of information backward in time.
You are missing the point. Newcomb’s problem, and other problems involving Omega, are unit tests for mathematical formalizations of decision theory. When a decision theory gets a contrived problem wrong, we don’t care because that scenario might appear in real life, but because it demonstrates that the math is wrong in a way that might make it subtly wrong on other problems, too.
I think you are missing the point. Newcomb’s problem is equivalent to dividing by zero. Decision theories aren’t supposed to behave well when abused in this way. If they behave badly on this problem, maybe it is the fault of the problem rather than the fault of the theory.
If someone can present a more robust decision theory, UDT or TDT, or whatever, which handles all the well formed problems just as well as standard game theory, and also handles the ill-formed problems like Newcomb in accord with EY’s intuitions, then I think that is great. I look forward to reading the papers and textbooks explaining that decision theory. But until they have gone through at least some serious process of peer review, please forgive me if I dismiss them as just so much woo and/or vaporware.
Incidentally, I specified “EY’s intuitions” rather than “correctness” as the criterion of success, because unless Omega actually appears and submits to a series of empirical tests, I can’t imagine a more respectable empirical criterion.
IMO, you haven’t made a case for that—and few here agree with you.
If you really think randomness is an issue, imagine a deterministic program facing the problem, with no good source of randomness to hand.
No, randomness is kind of a red herring. I shouldn’t have brought it up.
At one point I thought I had a kind of Dutch Book argument against Omega—if he could predict some future “random” event which I intended to use in conjunction with a mixed strategy, then I should be able to profit by making side bets “hedging” my choice with respect to Omega. But when I looked more carefully, it didn’t work.
Yay: honesty points!
True prisoner’s dilemma. Also, the prisoner’s dilemma generally. Newcomb just exemplifies the theme.
I don’t understand. You are answering my “show me”? Standard game theory says to defect in both PD and TPD. You have a revisionist decision theory that does better?
TDT does better, yes. My apologies; I’d forgotten the manuscript hasn’t yet been released to the public. It should be soon, I think; it’s been in the review process for a while. If for some reason Eliezer changed his mind and decided not to publish it then I’d be somewhat surprised. I’m guessing he’s nervous because it’s his first opportunity to show academia that he’s a real researcher and not just a somewhat bright autodidact.
There was a decision theory workshop a few months ago and a bunch of decision theorists are still working on solving the comparably much harder problems that were introduced at that time. Decision theory is still unsolved but UDT/TDT/ADT/XDT are a lot closer to solving it than the ancient CDT/EDT/SDT.
At the risk of looking stupid:
What are ADT and XDT?
For that matter, what’s SDT?
On the assumption that SDT stands for Sequential Decision Theory, I would like to take a shot at explaining this one, as well as at clarifying the relationship among CDT, EDT, and SDT. Everyone feel free to amend and extend my remarks.
Start with simple Bayesian updating. This is a theory of knowing, not a theory of acting. It helps you to know about the world, but doesn’t tell you what to do with your knowlege (other than to get more knowlege). There are two ways you can go from here: SDT and EDT.
SDT is basically game theory as developed by Seldin and Harsanyi. It adds agents, actions, and preferences to the world of propositions which exists in simple Bayesianism. Given the preferences of each agent regarding the propositions, and the agents’ beliefs about the effects which their actions have on the truth or falshood of propositions regarding which they have preferences, SDT advises each agent on their choice of actions. It is “sequential” because the decisions have to be considered in strict temporal order. For example, in the Parfit’s hitchhiker problem, both the hitchhiker and the motorist probably wish that the hitchhiker decision to pay $100 could be made before the motorist decision whether to offer a ride. But, in SDT, the decisions can not be made in this reverse order. By the same token, you cannot observe the future before deciding in the present.
If at least some of the agents in SDT believe that some of the other agents are rational, then you have game theory and things can get complicated. On the other hand, if you have only one agent, or if none of the agents believe that the others are rational, then you have classical decision theory which goes back to Wald (1939).
EDT is a variant of single-agent SDT due to Richard Jeffrey (1960s). In it, actions are treated just like any other proposition, except that some agents can make decisions that set action-propositions to be either true or false. The most interesting thing about EDT is that it is relatively “timeless”. That is, if X is an action, and A is an agent, then (A does X) might be thought of as a proposition. Using ordinary propositional logic, you can build and reason with compound propositions such as P → (A does X), (A does X)&Q->P, or (A does X)->(B does Y). The “timeless” aspect to this is that “A does X” is interpreted as “Either A did X, or is currently doing X, or will do X; I don’t really care about when it happens”.
The thing that makes EDT into a decision theory is the rule which says roughly “Act so as to make your preferred propositions true. If EDT worked as well as SDT, it would definitely be considered better, if only because of Ockham’s razor. It is an extremely elegant and simple theory. And it does work remarkably well. The most famous case where it doesn’t work (at least according to the SDT fans) is Newcomb’s problem. SDT says to two-box (because your decision cannot affect Omega’s already frozen-in-time decision). EDT says to one box (because it can’t even notice that the causality goes the wrong way). SDT and EDT also disagree regarding the Hitchhiker.
CDT is an attempt to improve on both SDT and EDT. It seems to be a work in progress. There are two variants out there right now—one built by philosophers and the other primarily the work of economist Judea Pearl. (I think I prefer Pearl’s version.) CDT helps to clarify the relationship between causation and correlation in Bayesian epistemology (i.e. learning). It also clarifies the relationship between action-based propositions (which are modeled in both SDT and EDT as somehow getting their truth value from the free will of the agents, and other propositions which get their truth value from the laws of physics. In CDT (Pearl’s version, at least) an action can be both free and determined—the flexibility reminds me of the compatibilist dissolution of the free will question which is suggested by the LW sequences.
I don’t know whether that summary answers the question you wanted answered, but I’m pretty sure the corrections I am likely to receive will answer the questions I want answered. :)
[Edit: corrected typos]
I think ADT is only described on Vladimir Nesov’s blog (if there) and XDT nowhere findable. ADT stands for Ambient Decision Theory. Unfortunately there’s no comprehensive and easy summary of any of the modern decision theories anywhere. Hopefully Eliezer publishes his TDT manuscript soon.
I coined the name XDT here. I think Anna Salamon and Steve Rayhawk had come up with essentially the same idea prior to that (and have explored its implications more deeply, but not in published form).
Thanks. I couldn’t find any references to ADT on Vladimir Nesov’s blog but I only had a quick scan so maybe I missed it, will have a better look later. And I can now remember that series of comments on XDT but my mind didn’t connect to it, thanks for the link.
DT list, nothing on the blog. Hopefully I’ll write up the current variant (which is conceptually somewhat different) in the near future.
Wow. I didn’t realise Eliezer had decided to actually release something formally. My recollection was that he was refusing to work on it unless someone promised him a PhD.
Does better how? By cooperating? By achieving an reverse-Omega-like stance and somehow constraining the other player to cooperate, conditionally on cooperating ourselves? I am completely mystified. I guess I will have to wait for the paper(s).
I don’t think there are any papers. There’s only this ramble:
http://lesswrong.com/lw/15z/ingredients_of_timeless_decision_theory/
As I said, I think your correspondents are in rather a muddle—and are discussing a completely different and rather esoteric PD case—where the agents can see and verify each other’s source code.
Thanks for the link. It was definitely telegraphic, but I think I got a pretty good notion where he is coming from with this, and also a bit about where he is going. I’m sure you remember the old days back at sci.bio.evolution talking about the various complications with the gene-level view of selection and Hamilton’s rule. Well, give another read to EY’s einsatz explanation of TDT:
Does that remind you of anything? “As you are deciding how the expression of you as a gene is going to affect the organism, remember to take into account that you are deciding for all of the members of your gene clone, and that changing the expression of your clone in other organisms is going to have an impact on the fitness of your own containing organism.” Now that is really cool. For the first time I begin to see how different decision theories might be appropriate for different meanings of the term “rational agent”.
I can’t claim to have understood everything EY wrote in that sketch, but I did imagine that I understood his concerns regarding “contrafactual surgery”. I want to get a hold of a preprint of the paper, when it is ready.
I think your correspondents are in rather a muddle—and are discussing a completely different and rather esoteric PD case—where the agents can see and verify each other’s source code. In which case, C-C is perfectly possible.
You have been given at least one such example in this thread and even had you not the process of taking an idealised problem and creating more mundane example should be one you are familiar with if you are as well versed in the literature as you claim.
Where was I given such an example? The only example I saw was of an unreliable Omega, an Omega who only gets it right 90% of the time.
If that is the example you mean, then (1) I agree that it adds unnecessary complexity by bringing in irrelevant considerations, and (2) I claim it is still f’ing impossible.
0 and 1 are not probabilities. I certainly don’t have a prior of 0 that Omega’s existence is impossible; he’s not defined in a contradictory fashion, and even if he was I harbor the tiniest bit of doubt that I’m wrong about how contradictions work.
I am using sloppy language here, perhaps. But to illustrate my usage, I claim that the probability that 2+2=4 is 1. And that p(2+2=5)=0.
If you were a Bayesian and assigned 0 probability to 2+2=5, you’d be in unrecoverable epistemic trouble if you turned out to be wrong about that. See How to convince me 2+2=3.
EY to the contrary, I remain smug in my evaluation p(2+2=5)=0. Of all the evidences that Eliezer offered, the only one to convince me was the one which demonstrated to me that I was confused about the meaning of the digit 5. Yes, by Cromwell’s rule, I think it possible I might be mistaken about how to count. “1, 2, 3, 5, 6, 4, 7”, I recite to myself. “Yes, I had been wrong about that. Thanks for correcting me.”
I might then write down p(Eliezer Yupkowski is the guru of Less Wrong)=0.999999999. Once again, I would be mistaken. It is “Yudkowski”, not “Yupkowski”) But in neither case am I in unrecoverable epistemic trouble. Those were typos. Correcting them is a simple search-and-replace, not a Bayesian updating. Or so I understand.
It’s Yudkowsky. Might want to update your general confidence evaluations.
Yudkowsky, in fact.
If you run out of material, here’s an academic paper, that claims to resolve many of the same problems as are being addressed on this site:
“DISPOSITION-BASED DECISION THEORY”
http://www.justin-fisher.com/papers/DBDT.pdf
CODT (Cop Out Decision Theory) : In which you precommit to every beneficial precommitment.
This Omega is not impossible.
It says: “Omega has been correct on each of 100 observed occasions so far”.
Not particularly hard—if you pick on decision theorists who had previously publicly expressed an opinion on the subject.
Ah! So I need to assign priors to three hypotheses. (1) Omega is a magician (i.e. illusion artist) (2) Omega had bribed people to lie about his past success. (3) He is what he claims.
So I assign a prior of zero probability to hypothesis #3, and cheerfully one-box using everyday decision theory.
First: http://lesswrong.com/lw/mp/0_and_1_are_not_probabilities/
You don’t seem to be entering into the spirit of the problem. You are “supposed” to reach the conclusion that there’s a good chance that Omega can predict your actions in this domain pretty well—from what he knows about you—after reading the premise of the problem.
If you think that’s not a practical possibility, then I recommend that you imagine yourself as a deterministic robot—where such a scenario becomes more believable—and then try the problem again.
If I imagine myself as a deterministic robot, who knows that he is a deterministic robot, I am no longer able to maintain the illusion that I care about this problem.
Do you think you aren’t a deterministic robot? Or that you are, but you don’t know it?
It is a quantum universe. So I would say that I am a stochastic robot. And Omega cannot predict my future actions.
...then you need to imagine you made the robot, it is meeting Omega on your behalf—and that it then gives you all its winnings.
I like this version! Now the answer seems quite obvious.
In this case, I would design the robot to be a one-boxer. And I would harbour the secret hope that a stray cosmic ray will cause the robot to pick both boxes anyway.
Yes—but you would still give its skull a lead-lining—and make use of redundancy to produce reliability...
Agreed.
For what it’s worth, I have written programs that cooperate on the prisoner’s dilemma if and only if their opponent will cooperate, without caring about the opponent’s rituals of cognition, only about his behaviour.
Unfortunately, this margin is too small to contain them, I mean, they’re not ready for prime time. I’ll probably write up a post on that in the near future.