What I meant was “if it were, that would require a large number of (I would expect) fairly intelligent mathematicians to have made an egregiously dumb mistake, on the order of an engineer modelling a 747 as made of cheese”. Does that seem likely to you? The principle of charity says “don’t assume someone is stupid so you can call them wrong”.
Yes, actually, they do seem have to have made an egregiously dumb mistake. People think EDT is dumb because it is dumb. Full stop.
The confusion is that sometimes when people talk about EDT, they are talking about the empirical group of “EDTers”. EDTers aren’t dumb enough to actually use the math of EDT. A “non-strawman EDT” is CDT. (If it wasn’t, how could the answers always be the same?) The point of math, though, is that you can’t strawman it; the math is what it is. Making decisions based on the conditional probabilities that resulted from observing that action historically is dumb, EDT makes decisions based on conditional probabilities, therefore EDT is dumb.
If it wasn’t, how could the answers always be the same?
They’re not...? EDT one-boxes on Newcomb’s and smokes (EDIT: doesn’t smoke) on the smoking lesion (unless the tickle defense actually works or something). Of course, it also two-boxes on transparent Newcomb’s, so it’s still a dumb theory, but it’s not that dumb.
How else should I interpret “I would expect (a particular non-strawman version of) EDT’s answer to be precisely the same as CDT’s answer”?
What I said was
Regardless, since there is nothing weird going on here, I would expect (a particular non-strawman version of) EDT’s answer to be precisely the same as CDT’s answer, because “agent’s action” has no common causes with the relevant outcomes
Meaning that in this particular situation (where there aren’t any omniscient predictors or mysterious correlations), the decision is the same. I didn’t mean they were the same generally.
Huh? EDT doesn’t smoke on the smoking lesion, because P(cancer|smoking)>P(cancer|!smoking).
Okay. Do you have a mathematical description of whether they differ, or is it a “I know it when I see it” sort of description? What makes a correlation mysterious?
I’m still having trouble imagining what a “non-strawman” EDT looks like mathematically, except for what I’m calling EDT+Intuition, which is people implicitly calculating probabilities using CDT and then using those probabilities to feed into EDT (in which case they’re only using it for expected value calculation, which CDT can do just as easily). It sounds to me like someone insisting that a “non-strawman” formula for x squared is x cubed.
A first try at formalising it would amount to “build a causal graph including EDT-agent’s-decision-now as a node, and calculated expected utilities using P(utility | agent=action, observations)”.
For example, for your average boring everyday situation, such as noticing a $5 note on the ground and thinking about whether to pick it up, the graph is (do I see $5 on the ground) --> (do I try to pick it up) --> (outcome). To arrive at a decision, you calculate the expected utilities using P(utility | pick it up, observation=$5) vs P(utility | don't pick it up, observation=$5). Note that conditioning on both observations and your action breaks the correlation expressed by the first link of the graph, resulting in this being equivalent to CDT in this situation. Also conveniently this makes P(action | I see $5) not matter, even though this is technically a necessary component to have a complete graph.
To be actually realistic you would need to include a lot of other stuff in the graph, such as everything else you’ve ever observed, and (agent's state 5 minutes ago) as causes of the current action (do I try to pick it up). But all of these can either be ignored (in the case of irrelevant observations) or marginalised out without effect (in the case of unobserved causes that we don’t know affect the outcome in any particular direction).
Next take an interesting case like Newcomb’s. The graph is something like the below:
We don’t know whether agent-5-minutes-ago was the sort that would make omega fill both boxes or not (so it’s not an observation), but we do know that there’s a direct correlation between that and our one-boxing. So when calculating P(utility|one-box), which implicitly involves marginalising over (agent-5-minutes-ago) and (omega fills boxes) we see that the case where (agent-5-minutes-ago)=one-box and (omega fills boxes)=both dominates, while the opposite case dominates for P(utility|two-box), so one-boxing has a higher utility.
A first try at formalising it would amount to “build a causal graph
Are we still talking about EDT? Why call it that?
(I do think that a good decision theory starts off with “build a causal graph,” but I think that decision theory already exists, and is CDT, so there’s no need to invent it again.)
I don’t think that formalization of Newcomb’s works, or at least you should flip the arrows. I think these are the formalizations in which perfect prediction makes sense:
You’re deciding which agent to be 5 minutes ago- and your decision is foreordained based on that. (This is the ‘submit source code to a game’ option like here.)
There is a causal arrow from your decision to Omega filling the boxes. (This is the Laplace’s Demon option, where the universe is deterministic and Omega, even though it’s in the past, is fully able to perceive the future. This is also your graph if you flip the ‘one-box?’ to ‘agent 5 minutes ago’ arrow.)
In both of these causal graphs, CDT suggests one-boxing. (I think that CDTers who two-box build the wrong causal graph from the problem description.)
It’s not like only CDTers are allowed to use causal graphs. You can call them “bayes nets” if the word “causal” seems too icky.
Joking aside, it’s called EDT because it doesn’t use do(·). We’re just using regular boring old conditional probabilities on the obvious formalisation of the problem.
As for reversing the arrows… I don’t think it’s entirely trivial to justify causal arrows that go backward in time. You can probably do it, with some kind of notion of “logical causality” or something. In fact, you could construct a more abstract network with “what this decision theory recommends” (a mathematical fact) as an ancestor node of both omega’s predictions and the agent itself. If you optimize the resulting utility over various values of the decision theory node I imagine you’d end up with something analogous to Wei Dei’s UDT (or was it ADT?). The decision theory node can be set as a parent of anywhere that decision theory is implemented in the universe, which was one of the main ideas of ADT.
I’m not sure if that could really be called “causal” decision theory any more though.
You can call them “bayes nets” if the word “causal” seems too icky.
Ugh. A Bayesian network is not a causal model. I am going to have to exit this, I am finding having to explain the same things over and over again very frustrating :(. From what I could tell following this thread you subscribe to the notion that there is no difference between EDT and CDT. That’s fine, I guess, but it’s a very exotic view of decision theories, to put to mildly. It just seems like a bizarre face-saving maneuver on behalf of EDT.
I have a little bit of unsolicited advice (which I know is dangerous to do), please do not view this as a status play: read the bit of Pearl’s book where he discusses the difference between a Bayesian network, a causal Bayesian network, and a non-parametric structural equation model. This may also make it clear what the crucial difference between EDT and CDT is.
Joking aside, it’s called EDT because it doesn’t use do(·). We’re just using regular boring old conditional probabilities on the obvious formalisation of the problem.
The central disagreement between EDT and CDT is whether one should use conditionals or counterfactuals. Counterfactuals are much stronger, and so I don’t see the argument for using conditionals.
In particular, if you’re just representing the joint probability distribution as a Bayes net, you don’t have any of the new information that a causal graph provides you. In particular, you cannot tell the difference between observations and interventions, which leads to all of the silliness of normal EDT. The do() function is a feature.
In causal graphs, information does not flow backwards across arrows in the presence of an intervention. (This is the difference between counterfactuals and conditionals.) If I make a decision now, it shouldn’t impact things that might cause other people to make that decision. (Getting out of bed does not affect the time of day, even though the time of day affects getting out of bed.) When the decision “one-box?” impacts the node “5 minutes ago,” it’s part of that class of causal errors, as when someone hopes that staying in bed will make it still be morning when they decide to get out of bed. Your use of the graph as you described it has no internal mechanism to avoid that, and so would seek to “manage the news” in other situations.
This is why EDT looks so broken, and why IlyaShpitser in particular is so interested in seeing the EDT method actually worked out. It’s like the painter who insisted that he could get yellow by mixing red and white paint. In CDT, you get the causal separation between observations and interventions with the do() operator, but in EDT, you need a tube of yellow to ‘sharpen it up’.
As for reversing the arrows… I don’t think it’s entirely trivial to justify causal arrows that go backward in time. You can probably do it, with some kind of notion of “logical causality” or something.
I think that perfect prediction is functionally equivalent to causality flowing backwards in time, and don’t think it’s possible to construct a counterexample. I agree it doesn’t happen in the real world, but in the hypothetical world of Newcomb’s Problem, that’s the way things are, and so resisting that point is fighting the hypothetical. (If time travel were possible in our world, would the fact that Omega had made its prediction and left be at all decision-theoretically compelling? I don’t think so, and so I strongly suspect that two-boxing is primarily caused by misbehaving temporal intuitions.)
I’m not actually advocating EDT. After all, it two-boxes on transparent Newcomb’s, which is a clear mistake. I’m just trying to explain how it’s not as bad as it seems to be. For example:
as when someone hopes that staying in bed will make it still be morning when they decide to get out of bed. Your use of the graph as you described it has no internal mechanism to avoid that, and so would seek to “manage the news” in other situations.
The only reason there is normally a correlation between getting out of bed and “morning time” is because people decide whether to get out of bed based on the clock, sunrise, etc. I do not think that EDT would “stay in bed in order to make it still be morning” because:
If you’ve already looked at the clock, there’s no hope for you anyway. It’s already a practical certainty that it won’t be morning any more if you stay in bed and get up later. This screens off any possible effect that deciding could have. (Granted, there’s a tiny remnant effect for “what if the laws of physics changed and time doesn’t necessarily work the same ways as I’m used to”. But your samples that suggest a correlation don’t cover that case, and even if they did, they would show that people who stayed in bed hoping it would make morning last longer typically missed their early-morning commute.)
Or, if you’ve somehow managed to remain ignorant as to the time, the causal connection between “getting out of bed” and “morning” is broken, because you can’t possibly be deciding whether to get out of bed based on the time. So the graph doesn’t even have a connection between the two nodes, and staying in bed does nothing.
In general “managing the news” doesn’t work because “managing” is a very different behaviour to which simply doesn’t correlate to the information in the same way. EDT, done right, is aware of that.
I’m not actually advocating EDT. After all, it two-boxes on transparent Newcomb’s, which is a clear mistake. I’m just trying to explain how it’s not as bad as it seems to be.
I don’t see why you think that EDT isn’t “as bad as it seems to be,” yet. I see lots of verbal explanations of why EDT wouldn’t do X, but what I’m looking for is a mathematical reason why you think the language of conditionals is as strong as the language of counterfactuals, or why a decision theory that only operates in conditionals will not need human guidance (which uses counterfactuals) to avoid getting counterfactual questions wrong.
Are you under the impression that I think EDT is The Best Thing Ever or something???
EDT gets stuff wrong. It gets transparent newcomb’s wrong. It does not, however kill puppies hoping that this means that there is some sane reason to be killing puppies (because obviously people only ever kill puppies for a good reason).
The reason for this should be obvious [hint: killing puppies in the hope that there’s a good reason does not correlate with having good reasons].
I’ve explained this multiple times. I’m not going to play “person from the non-CDT tribe” any more if you still can’t understand. I don’t even like EDT...
ETA:
mathematical reason
Remember when I gave you a mathematical formalism and you ignored it saying “your arrows are the wrong way around” and “CDT is better, by the way”? That’s frustrating. This isn’t a competition.
Are you under the impression that I think EDT is The Best Thing Ever or something???
No, I think you think it’s not as bad as it seems to be. Things not being as they seem is worth investigating, especially about technical topics where additional insight could pay serious dividends.
I’m not going to play “person from the non-CDT tribe” any more if you still can’t understand.
I apologize for any harshness. I don’t think a tribal lens is useful for this discussion, though; friends can disagree stringently on technical issues. I think that as an epistemic principle it’s worth pursuing such disagreements, even though disagreements in other situations might be a signal of enmity.
Remember when I gave you a mathematical formalism and you ignored it saying “your arrows are the wrong way around” and “CDT is better, by the way”?
It’s not clear to me why you think I ignored your formalism; I needed to understand it to respond how I did.
My view is that there are two options for your formalism:
The formalism should be viewed as a causal graph, and decisions as the do() operator, in which case the decision theory is CDT and the formalism does not capture Newcomb’s problem because Omega is not a perfect predictor. Your formalism describes the situation where you want to be a one-boxer when scanned, and then switch to two-boxing after Omega has left, without Omega knowing that you’ll switch. If this is impossible, you are choosing who to be 5 minutes ago, not your decision now (or your decision now causes who you were 5 minutes ago), and if this is possible, then Omega isn’t a perfect predictor.
The formalism should be viewed as a factorization of a joint probability distribution, and decisions as observations, in which case the decision theory is EDT and the formalism will lead to “managing the news” in other situations. (If your formalism is a causal graph but decisions are observations, then the ‘causal’ part of the graph is unused, and I don’t see a difference between it and a factorization of a joint probability distribution.)
To justify the second view: I do not see a structural difference between
We don’t know whether agent-5-minutes-ago was the sort that would make omega fill both boxes or not (so it’s not an observation), but we do know that there’s a direct correlation between that and our one-boxing.
and
We don’t know what time it is (so it’s not an observation), but we do know that there’s a direct correlation between that and our getting out of bed.
whereas with the do() operator, the structural difference is clear (interventions do not propagate backwards across causal arrows). If you want to use language like “the causal connection between “getting out of bed” and “morning” is broken,” that language makes sense with the do() operator but doesn’t make sense with conditioning. How are you supposed to know which correlations are broken, and which aren’t, without using the do() operator?
To justify the second view: I do not see a structural difference between
We don’t know whether agent-5-minutes-ago was the sort that would make omega fill both boxes or not (so it’s not an observation), but we do know that there’s a direct correlation between that and our one-boxing.
and
We don’t know what time it is (so it’s not an observation), but we do know that there’s a direct correlation between that and our getting out of bed.
The difference is that in the first case the diamond graph is still accurate, because our dataset or other evidence that we construct the graph from says that there’s a correlation between 5-minutes-ago and one-box even when the agent doesn’t know the state of agent-5-minutes-ago.
In the second case there should be no connection from time to get-out-of-bed, because we haven’t observed the time, and all our samples which would otherwise suggest a correlation involve an agent who has observed the time, and decides whether to get out of bed based on that, so they’re inapplicable to this situation. More strongly, we know that the supposed correlation is mediated by “looking at the clock and deciding to get up if it’s past 8am”, which cannot happen here. There is no correlation until we observe the time.
The difference is that in the first case the diamond graph is still accurate
My understanding of this response is “the structural difference between the situations is that the causal graphs are different for the two situations.” I agree with that statement, but I think you have the graphs flipped around; I think the causal graph you drew describes the get-out-of-bed problem, and not Newcomb’s problem. I think the following causal graph fits the “get out of bed?” situation:
Time → I get out of bed → Late, Time → Boss gets out of bed → Late
(I’ve added the second path to make the graphs exactly analogous; suppose I’m only late for work if I arrive after my boss, and if I manage to change what time it is, I’ll change when my boss gets to work because I’ve changed what time it is.)
This has the same form as the Newcomb’s causal graph that you made; only the names have changed. In that situation, you asserted that the direct correlation between “one-box?” and “agent 5 minutes ago” was strong and relevant to our decision, even though we hadn’t observed “agent 5 minutes ago,” and historical agents who played Omega’s game hadn’t observed “agent 5 minutes ago” when they played. I assert that there’s a direct correlation between Time and I-get-out-of-bed which doesn’t depend on observing Time. (The assumption that the correlation between getting out of bed and the time is mediated by looking at the clock is your addition, and doesn’t need to be true; let’s assume the least convenient possible world where it isn’t mediated by that. I can think of a few examples and I’m sure you could as well.)
And so when we calculate P(Late|I-get-out-of-bed) using the method you recommended for Newcomb’s in the last paragraph of this comment, we implicitly marginalize over Time and Boss-gets-out-of-bed, and notice that when we choose to stay in bed, this increases the probability that it’s early, which increases the probability that our boss chooses to stay in bed!
For the two graphs- which only differ in the names of the nodes- to give different mathematical results, we would need to have our math pay attention to the names of the nodes. This is a fatal flaw, because it means we need to either teach our math how to read English, or hand-guide it every time we want to use it.
This automatic understanding of causality is what the mathematical formalism of interventions is good for: while observations flow back up causal chains (seeing my partner get out of bed is evidence about the time, or looking at a clock is evidence about the time), interventions don’t (seeing myself get out of bed isn’t evidence about the time, or adjusting my clock isn’t evidence about the time). All the English and physics surrounding these two problems can be encapsulated in the direction of the causal arrows in the graph, and then the math can proceed automatically.
This requires that different problems have different graphs: the “get out of bed” problem has the graph I describe here, which is identical to your Newcomb’s graph, and the Newcomb’s graph has the arrow pointing from one-box? to agent-5-minutes-ago, rather than the other way around, like I suggested here. With the bed graph I recommend, choosing to stay in bed cannot affect your boss’s arrival time, and with the Newcomb’s graph I recommend, your decision to one-box can affect Omega’s filling of the boxes. We get the sane results- get out of bed and one-box.
This way, all the exoticism of the problem goes into constructing the graph (a graph where causal arrows point backwards in time is exotic!), not into the math of what to do with a graph.
I assert that there’s a direct correlation between Time and I-get-out-of-bed which doesn’t depend on observing Time. (The assumption that the correlation between getting out of bed and the time is mediated by looking at the clock is your addition, and doesn’t need to be true; let’s assume the least convenient possible world where it isn’t mediated by that. I can think of a few examples and I’m sure you could as well.)
You can turn any ordinary situation into the smoking lesion by postulating mysterious correlations rather than straightforward correlations that work by conscious decisions based on observations. Did you have any realistic examples in mind?
Yes. Suppose that the person always goes to sleep at the same time, and wakes up after a random interval. In the dark of their bedroom (blackout curtains and no clock), they decide whether or not to go back to sleep or get up. Historically, the later in the morning it is, the more likely they are to get up. To make the historical record analogous to Newcomb’s, we might postulate that historically, they have always decided to go back to sleep before 7 AM, and always decided to get up after 7 AM, and the boss’s alarm is set to wake him up at 7 AM. This is not a very realistic postulation, as a stochastic relationship between the two is more realistic, but the parameters of the factorization are not related to the structure of the causal graph (and Newcomb’s isn’t very realistic either).
You can turn any ordinary situation into the smoking lesion by postulating mysterious correlations rather than straightforward correlations that work by conscious decisions based on observations.
It’s not obvious to me what you mean by “mysterious correlations” and “straightforward correlations.” Correlations are statistical objects that either exist or don’t, and I don’t know what conscious decision based on observations you’re referring to in the smoking lesion problem. What makes the smoking lesion problem a problem is that the lesions are unobserved.
For example, in an Israeli study, parole decisions are correlated with the time since the parole board last took a break. Is that correlation mysterious, or straightforward? No one familiar with the system (board members, lawyers, etc.) expected the effects the study revealed, but there are plausible explanations for the effect (mental fatigue, low blood sugar, declining mood, all of which are replenished by a meal break).
It might be that by ‘mysterious correlations’ you mean ‘correlations without an obvious underlying causal mechanism’, and by ‘straightforward correlations’ you mean ‘correlations with an obvious underlying causal mechanism.’ It’s not clear to me what the value of that distinction is. Neither joint probability distributions nor causal graphs do not require that the correlations or causal arrows be labeled.
On the meta level, though, I don’t think it’s productive to fight the hypothetical. I strongly recommend reading The Least Convenient Possible World.
It’s not obvious to me what you mean by “mysterious correlations” and “straightforward correlations.” Correlations are statistical objects that either exist or don’t, and I don’t know what conscious decision based on observations you’re referring to in the smoking lesion problem. What makes the smoking lesion problem a problem is that the lesions are unobserved.
Well, the correlations in the smoking lesion problem are mysterious because they aren’t caused by agents observing lesion|no-lesion and deciding whether to smoke based on that. They are mysterious because it is simply postulated that “the lesion causes smoking without being observed” without any explanation of how, and it is generally assumed that the correlation somehow still applies when you’re deciding what to do using EDT, which I personally have some doubt about (EDT decides what to do based only on preferences and observations, so how can its output be correlated to anything else?).
Straightforward correlations are those where, for example, people go out with an umbrella if they see rain clouds forming. The correlation is created by straightforward decision-making based on observations. Simple statistical reasoning suggests that you only have reason to expect these correlations to hold for an EDT agent if the EDT agent makes the same decisions in the same situations. Furthermore, these correlations tend to pose no problem for EDT because the only time an EDT agent is in a position to take an action correlated to some observation in this way (“I observe rain clouds, should I take my umbrella?”), they must have already observed the correlate (“rain clouds”), so EDT makes no attempt to influence it (“whether or not I take my umbrella, I know there are rain clouds already”) .
Returning to the smoking lesion problem, there are a few ways of making the mystery go away. You can suppose that the lesion works by making you smoke even after you (consciously) decide to do something else. In this case the decision of the EDT agent isn’t actually smoke | don't-smoke, but rather you get to decide a parameter of something else that determines whether you smoke. This makes the lesion not actually a cause of your decision, so you choose-to-smoke, obviously.
Alternatively, I was going to analyse the situation where the lesion makes you want to smoke (by altering your decision theory/preferences), but it made my head hurt. I anticipate that EDT wouldn’t smoke in that situation iff you can somehow remain ignorant of your decision or utility function even while implementing EDT, but I can’t be sure.
Basically, the causal reasons behind your data (why do people always get up after 7AM?) matter, because they determine what kind of causal graph you can infer for the situation with an EDT agent with some given set of observations, as opposed to whatever agents are in the dataset.
Postscript regarding LCPW: If I’m trying to argue that EDT doesn’t normally break, then presenting a situation where it does break isn’t necessarily proper LCPW. Because I never argued that it always did the right thing (which would require me to handle edge cases).
it is simply postulated that “the lesion causes smoking without being observed” without any explanation of how
No mathematical decision theory requires verbal explanations to be part of the model that it operates on. (It’s true that when learning a causal model from data, you need causal assumptions; but when a problem provides the model rather than the data, this is not necessary.)
it is generally assumed that the correlation somehow still applies when you’re deciding what to do using EDT, which I personally have some doubt about
You have doubt that this is how EDT, as a mathematical algorithm, operates, or you have some doubt that this is a wise way to construct a decision-making algorithm?
If the second, this is why I think EDT is a subpar decision theory. It sees the world as a joint probability distribution, and does not have the ability to distinguish correlation and causation, which means it cannot know whether or not a correlation applies for any particular action (and so assumes that all do).
If the first, I’m not sure how to clear up your confusion. There is a mindset that programming cultivates, which is that the system does exactly what you tell it to, with the corollary that your intentions have no weight.
If I’m trying to argue that EDT doesn’t normally break, then presenting a situation where it does break isn’t necessarily proper LCPW.
The trouble with LCPW is that it’s asymmetric; Eliezer claims that the LCPW is the one where his friend has to face a moral question, and Eliezer’s friend might claim that the LCPW is the one where Eliezer has to face a practical problem.
The way to break the asymmetry is to try to find the most informative comparison. If the hypothetical has been fought, then we learn nothing about morality, because there is no moral problem. If the hypothetical is accepted despite faults, then we learn quite a bit about morality.
The issues with EDT might require ‘edge cases’ to make obvious, but in the same way that the issues with Newtonian dynamics might require ‘edge cases’ to make obvious.
No mathematical decision theory requires verbal explanations to be part of the model that it operates on. (It’s true that when learning a causal model from data, you need causal assumptions; but when a problem provides the model rather than the data, this is not necessary.)
What I’m saying is that the only way to solve any decision theory problem is to learn a causal model from data. It just doesn’t make sense to postulate particular correlations between an EDT agent’s decisions and other things before you even know what EDT decides! The only reason you get away with assuming graphs like lesion -> (CDT Agent) -> action for CDT is because the first thing CDT does when calculating a decision is break all connections to parents by means of do(...).
Take Jiro’s example. The lesion makes people jump into volcanoes. 100% of them, and no-one else. Furthermore, I’ll postulate that all of them are using decision theory “check if I have the lesion, if so, jump into a volcano, otherwise don’t”. Should you infer the causal graph lesion -> (EDT decision: jump?) -> die with a perfect correlation between lesion and jump? (Hint: no, that would be stupid, since we’re not using jump-based-on-lesion-decision-theory, we’re using EDT.)
There is a mindset that programming cultivates, which is that the system does exactly what you tell it to, with the corollary that your intentions have no weight.
In programming, we also say “garbage in, garbage out”. You are feeding EDT garbage input by giving it factually wrong joint probability distributions.
Ok, what about cases where there are multiple causal hypotheses that are observationally indistinguishable:
a → b → c
vs
a ← b ← c
Both models imply the same joint probability distribution p(a,b,c) with a single conditional independence (a independent of c given b) and cannot be told apart without experimentation. That is, you cannot call p(a,b,c) “factually wrong” because the correct causal model implies it. But the wrong causal model implies it too! To figure out which is which requires causal information. You can give it to EDT and it will work—but then it’s not EDT anymore.
I can give you a graph which implies the same independences as my HAART example but has a completely different causal structure, and the procedure you propose here:
will give the right answer in one case and the wrong answer in another.
The point is, EDT lacks a rich enough input language to avoid getting garbage inputs in lots of standard cases. Or, more precisely, EDT lacks a rich enough input languages to tell when input is garbage and when it isn’t. This is why EDT is a terrible decision theory.
What I’m saying is that the only way to solve any decision theory problem is to learn a causal model from data.
I think there are a couple of confusions this sentence highlights.
First, there are approaches to solving decision theory problems that don’t use causal models. Part of what has made this conversation challenging is that there are several different ways to represent the world- and so even if CDT is the best / natural one, it needs to be distinguished from other approaches. EDT is not CDT in disguise; the two are distinct formulas / approaches.
Second, there are good reasons to modularize the components of the decision theory, so that you can treat learning a model from data separately from making a decision given a model. An algorithm to turn models into decisions should be able to operate on an arbitrary model, where it sees a → b → c as isomorphic to Drunk → Fall → Death.
To tell an anecdote, when my decision analysis professor would teach that subject to petroleum engineers, he quickly learned not to use petroleum examples. Say something like “suppose the probability of striking oil by drilling a well here is 40%” and an engineer’s hand will shoot up, asking “what kind of rock is it?”. The kind of rock is useful for determining whether or not the probability is 40% or something else, but the question totally misses the point of what the professor is trying to teach. The primary example he uses is choosing a location for a party subject to the uncertainty of the weather.
It just doesn’t make sense to postulate particular correlations between an EDT agent’s decisions and other things before you even know what EDT decides!
I’m not sure how to interpret this sentence.
The way EDT operates is to perform the following three steps for each possible action in turn:
Assume that I saw myself doing X.
Perform a Bayesian update on this new evidence.
Calculate and record my utility.
It then chooses the possible action which had the highest calculated utility.
One interpretation is you saying that EDT doesn’t make sense, but I’m not sure I agree with what seems to be the stated reason. It looks to me like you’re saying “it doesn’t make sense to assume that you do X until you know what you decide!”, when I think that does make sense, but the problem is using that assumption as Bayesian evidence as if it were an observation.
The way EDT operates is to perform the following three steps for each possible action in turn:
Assume that I saw myself doing X.
Perform a Bayesian update on this new evidence.
Calculate and record my utility.
Ideal Bayesian updates assume logical omniscience, right? Including knowledge about logical fact of what EDT would do for any given input. If you know that you are an EDT agent, and condition on all of your past observations and also on the fact that you do X, but X is not in fact what EDT does given those inputs, then as an ideal Bayesian you will know that you’re conditioning on something impossible. More generally, what update you perform in step 2 depends on EDT’s input-output map, thus making the definition circular.
So, is EDT really underspecified? Or are you supposed to search for a fixed point of the circular definition, if there is one? Or does it use some method other than Bayes for the hypothetical update? Or does an EDT agent really break if it ever finds out its own decision algorithm? Or did I totally misunderstand?
Ideal Bayesian updates assume logical omniscience, right? Including knowledge about logical fact of what EDT would do for any given input.
Note that step 1 is “Assume that I saw myself doing X,” not “Assume that EDT outputs X as the optimal action.” I believe that excludes any contradictions along those lines. Does logical omniscience preclude imagining counterfactual worlds?
If I already know “I am EDT”, then “I saw myself doing X” does imply “EDT outputs X as the optimal action”. Logical omniscience doesn’t preclude imagining counterfactual worlds, but imagining counterfactual worlds is a different operation than performing Bayesian updates. CDT constructs counterfactuals by severing some of the edges in its causal graph and then assuming certain values for the nodes that no longer have any causes. TDT does too, except with a different graph and a different choice of edges to sever.
I don’t know how I can fail to communicate so consistently.
Yes, you can technically apply “EDT” to any causal model or (more generally) joint probability distribution containing a “EDT agent decision” node. But in practice this freedom is useless, because to derive an accurate model you generally need to take account of a) the fact that the agent is using EDT and b) any observations the agent does or does not make. To be clear, the input EDT requires is a probabilistic model describing the EDT agent’s situation (not describing historical data of “similar” situations).
There are people here trying to argue against EDT by taking a model describing historical data (such as people following dumb decision theories jumping into volcanoes) and feeding this model directly into EDT. Which is simply wrong. A model that describes the historical behaviour of agents using some other decision theory does not in general accurately describe an EDT agent in the same situation.
The fact that this egregious mistake looks perfectly normal is an artifact of the fact that CDT doesn’t care about causal parents of the “CDT decision” node.
I don’t know how I can fail to communicate so consistently.
I suspect it’s because what you are referring to as “EDT” is not what experts in the field use that technical term to mean.
nsheppard-EDT is, as far as I can tell, the second half of CDT. Take a causal model and use the do() operator to create the manipulated subgraph that would result taking possible action (as an intervention). Determine the joint probability distribution from the manipulated subgraph. Condition on observing that action with the joint probability distribution, and calculate the probabilistically-weighted mean utility of the possible outcomes. This is isomorphic to CDT, and so referring to it as EDT leads to confusion.
Here’s a modified version. Instead of a smoking lesion, there’s a “jump into active volcano lesion”. Furthermore, the correlation isn’t as puny as for the smoking lesion. 100% of people with this lesion jump into active volcanoes and die, and nobody else does.
Should you go jump into an active volcano?
Using a decision theory to figure out what decision you should make assumes that you’re capable of making a decision. “The lesion causes you to jump into an active volcano/smoke” and “you can choose whether to jump into an active volcano/smoke” are contradictory. Even “the lesion is correlated (at less than 100%) with jumping into an active volcano/smoking” and “you can choose whether to jump into an active volcano/smoke” are contradictory unless “is correlated with” involves some correlation for people who don’t use decision theory and no correlation for people who do.
Using a decision theory to figure out what decision you should make assumes that you’re capable of making a decision.
Agreed.
unless “is correlated with” involves some correlation for people who don’t use decision theory and no correlation for people who do.
Doesn’t this seem sort of realistic, actually? Decisions made with System 1 and System 2, to use Kahneman’s language, might have entirely different underlying algorithms. (There is some philosophical trouble about how far we can push the idea of an ‘intervention’, but I think for human-scale decisions there is a meaningful difference between interventions and observations such that CDT distinguishing between them is a feature.)
This maps onto an objection by proponents of EDT that the observational data might not be from people using EDT, and thus the correlation may disappear when EDT comes onto the stage. I think that objection proves too much- suppose all of our observational data on the health effects of jumping off cliffs comes from subjects who were not using EDT (suppose they were drunk). I don’t see a reason inside the decision theory for differentiating between the effects of EDT on the correlation between jumping off the cliff and the effects of EDT on the correlation between smoking and having the lesion.
These two situations correspond to two different causal structures—Drunk → Fall → Death and Smoke ← Lesion → Cancer—which could have the same joint probability distribution. The directionality of the arrow is something that CDT can make use of to tell that the two situations will respond differently to interventions at Drunk and Smoke: it is dangerous to be drunk around cliffs, but not to smoke (in this hypothetical world).
EDT cannot make use of those arrows. It just has Drunk—Fall—Death and Smoke—Lesion—Cancer (where it knows that the correlations between Drunk and Death are mediated by Fall, and the correlations between Smoke and Cancer are mediated by Lesion). If we suppose that adding an EDT node might mean that the correlation between Smoke and Lesion (and thus Cancer) might be mediated by EDT, then we must also suppose that adding an EDT node might mean that the correlation between Drunk and Fall (and thus Death) might be mediated by EDT.
(I should point out that the EDT node describes whether or not EDT was used to decide to drink, not to decide whether or not to fall off the cliff, by analogy of using EDT to decide whether or not to smoke, rather than deciding whether or not to have a lesion.)
there’s another CRUCIAL difference regarding the Newcombs problem: there’s always a chance you’re in a simulation being run by Omega. I think if you can account for that, it SHOULD patch most decent decision-theories up. I’m willing to be quite flexible in my understanding of which theories get patched up or not.
this has the BIG advantage of NOT requiring non-linear causality in the model-it just gives a flow from simulation->”real”world.
there’s always a chance you’re in a simulation being run by Omega. I think if you can account for that, it SHOULD patch most decent decision-theories up.
Yes, reflective consistency tends to make things better.
Yes, actually, they do seem have to have made an egregiously dumb mistake. People think EDT is dumb because it is dumb. Full stop.
The confusion is that sometimes when people talk about EDT, they are talking about the empirical group of “EDTers”. EDTers aren’t dumb enough to actually use the math of EDT. A “non-strawman EDT” is CDT. (If it wasn’t, how could the answers always be the same?) The point of math, though, is that you can’t strawman it; the math is what it is. Making decisions based on the conditional probabilities that resulted from observing that action historically is dumb, EDT makes decisions based on conditional probabilities, therefore EDT is dumb.
They’re not...? EDT one-boxes on Newcomb’s and smokes (EDIT: doesn’t smoke) on the smoking lesion (unless the tickle defense actually works or something). Of course, it also two-boxes on transparent Newcomb’s, so it’s still a dumb theory, but it’s not that dumb.
How else should I interpret “I would expect (a particular non-strawman version of) EDT’s answer to be precisely the same as CDT’s answer”?
Huh? EDT doesn’t smoke on the smoking lesion, because P(cancer|smoking)>P(cancer|!smoking).
What I said was
Meaning that in this particular situation (where there aren’t any omniscient predictors or mysterious correlations), the decision is the same. I didn’t mean they were the same generally.
Er, you’re right. I got mixed up there.
Okay. Do you have a mathematical description of whether they differ, or is it a “I know it when I see it” sort of description? What makes a correlation mysterious?
I’m still having trouble imagining what a “non-strawman” EDT looks like mathematically, except for what I’m calling EDT+Intuition, which is people implicitly calculating probabilities using CDT and then using those probabilities to feed into EDT (in which case they’re only using it for expected value calculation, which CDT can do just as easily). It sounds to me like someone insisting that a “non-strawman” formula for x squared is x cubed.
A first try at formalising it would amount to “build a causal graph including EDT-agent’s-decision-now as a node, and calculated expected utilities using P(utility | agent=action, observations)”.
For example, for your average boring everyday situation, such as noticing a $5 note on the ground and thinking about whether to pick it up, the graph is
(do I see $5 on the ground) --> (do I try to pick it up) --> (outcome). To arrive at a decision, you calculate the expected utilities usingP(utility | pick it up, observation=$5) vs P(utility | don't pick it up, observation=$5). Note that conditioning on both observations and your action breaks the correlation expressed by the first link of the graph, resulting in this being equivalent to CDT in this situation. Also conveniently this makesP(action | I see $5)not matter, even though this is technically a necessary component to have a complete graph.To be actually realistic you would need to include a lot of other stuff in the graph, such as everything else you’ve ever observed, and
(agent's state 5 minutes ago)as causes of the current action(do I try to pick it up). But all of these can either be ignored (in the case of irrelevant observations) or marginalised out without effect (in the case of unobserved causes that we don’t know affect the outcome in any particular direction).Next take an interesting case like Newcomb’s. The graph is something like the below:
We don’t know whether
agent-5-minutes-agowas the sort that would make omega fill both boxes or not (so it’s not an observation), but we do know that there’s a direct correlation between that and our one-boxing. So when calculatingP(utility|one-box), which implicitly involves marginalising over(agent-5-minutes-ago)and(omega fills boxes)we see that the case where(agent-5-minutes-ago)=one-boxand(omega fills boxes)=bothdominates, while the opposite case dominates forP(utility|two-box), so one-boxing has a higher utility.Are we still talking about EDT? Why call it that?
(I do think that a good decision theory starts off with “build a causal graph,” but I think that decision theory already exists, and is CDT, so there’s no need to invent it again.)
I don’t think that formalization of Newcomb’s works, or at least you should flip the arrows. I think these are the formalizations in which perfect prediction makes sense:
You’re deciding which agent to be 5 minutes ago- and your decision is foreordained based on that. (This is the ‘submit source code to a game’ option like here.)
There is a causal arrow from your decision to Omega filling the boxes. (This is the Laplace’s Demon option, where the universe is deterministic and Omega, even though it’s in the past, is fully able to perceive the future. This is also your graph if you flip the ‘one-box?’ to ‘agent 5 minutes ago’ arrow.)
In both of these causal graphs, CDT suggests one-boxing. (I think that CDTers who two-box build the wrong causal graph from the problem description.)
It’s not like only CDTers are allowed to use causal graphs. You can call them “bayes nets” if the word “causal” seems too icky.
Joking aside, it’s called EDT because it doesn’t use
do(·). We’re just using regular boring old conditional probabilities on the obvious formalisation of the problem.As for reversing the arrows… I don’t think it’s entirely trivial to justify causal arrows that go backward in time. You can probably do it, with some kind of notion of “logical causality” or something. In fact, you could construct a more abstract network with “what this decision theory recommends” (a mathematical fact) as an ancestor node of both omega’s predictions and the agent itself. If you optimize the resulting utility over various values of the decision theory node I imagine you’d end up with something analogous to Wei Dei’s UDT (or was it ADT?). The decision theory node can be set as a parent of anywhere that decision theory is implemented in the universe, which was one of the main ideas of ADT.
I’m not sure if that could really be called “causal” decision theory any more though.
Ugh. A Bayesian network is not a causal model. I am going to have to exit this, I am finding having to explain the same things over and over again very frustrating :(. From what I could tell following this thread you subscribe to the notion that there is no difference between EDT and CDT. That’s fine, I guess, but it’s a very exotic view of decision theories, to put to mildly. It just seems like a bizarre face-saving maneuver on behalf of EDT.
I have a little bit of unsolicited advice (which I know is dangerous to do), please do not view this as a status play: read the bit of Pearl’s book where he discusses the difference between a Bayesian network, a causal Bayesian network, and a non-parametric structural equation model. This may also make it clear what the crucial difference between EDT and CDT is.
Also read this if you have time: www.biostat.harvard.edu/robins/publications/wp100.pdf
This paper discusses what a causal model is very clearly (actually it discusses 4 separate causal models arranged in a hierarchy of “strength.”)
EDT one-boxes on newcomb’s. Also I am well aware that not all bayes nets are causal models.
The central disagreement between EDT and CDT is whether one should use conditionals or counterfactuals. Counterfactuals are much stronger, and so I don’t see the argument for using conditionals.
In particular, if you’re just representing the joint probability distribution as a Bayes net, you don’t have any of the new information that a causal graph provides you. In particular, you cannot tell the difference between observations and interventions, which leads to all of the silliness of normal EDT. The do() function is a feature.
In causal graphs, information does not flow backwards across arrows in the presence of an intervention. (This is the difference between counterfactuals and conditionals.) If I make a decision now, it shouldn’t impact things that might cause other people to make that decision. (Getting out of bed does not affect the time of day, even though the time of day affects getting out of bed.) When the decision “one-box?” impacts the node “5 minutes ago,” it’s part of that class of causal errors, as when someone hopes that staying in bed will make it still be morning when they decide to get out of bed. Your use of the graph as you described it has no internal mechanism to avoid that, and so would seek to “manage the news” in other situations.
This is why EDT looks so broken, and why IlyaShpitser in particular is so interested in seeing the EDT method actually worked out. It’s like the painter who insisted that he could get yellow by mixing red and white paint. In CDT, you get the causal separation between observations and interventions with the do() operator, but in EDT, you need a tube of yellow to ‘sharpen it up’.
I think that perfect prediction is functionally equivalent to causality flowing backwards in time, and don’t think it’s possible to construct a counterexample. I agree it doesn’t happen in the real world, but in the hypothetical world of Newcomb’s Problem, that’s the way things are, and so resisting that point is fighting the hypothetical. (If time travel were possible in our world, would the fact that Omega had made its prediction and left be at all decision-theoretically compelling? I don’t think so, and so I strongly suspect that two-boxing is primarily caused by misbehaving temporal intuitions.)
I’m not actually advocating EDT. After all, it two-boxes on transparent Newcomb’s, which is a clear mistake. I’m just trying to explain how it’s not as bad as it seems to be. For example:
The only reason there is normally a correlation between getting out of bed and “morning time” is because people decide whether to get out of bed based on the clock, sunrise, etc. I do not think that EDT would “stay in bed in order to make it still be morning” because:
If you’ve already looked at the clock, there’s no hope for you anyway. It’s already a practical certainty that it won’t be morning any more if you stay in bed and get up later. This screens off any possible effect that deciding could have. (Granted, there’s a tiny remnant effect for “what if the laws of physics changed and time doesn’t necessarily work the same ways as I’m used to”. But your samples that suggest a correlation don’t cover that case, and even if they did, they would show that people who stayed in bed hoping it would make morning last longer typically missed their early-morning commute.)
Or, if you’ve somehow managed to remain ignorant as to the time, the causal connection between “getting out of bed” and “morning” is broken, because you can’t possibly be deciding whether to get out of bed based on the time. So the graph doesn’t even have a connection between the two nodes, and staying in bed does nothing.
In general “managing the news” doesn’t work because “managing” is a very different behaviour to which simply doesn’t correlate to the information in the same way. EDT, done right, is aware of that.
I don’t see why you think that EDT isn’t “as bad as it seems to be,” yet. I see lots of verbal explanations of why EDT wouldn’t do X, but what I’m looking for is a mathematical reason why you think the language of conditionals is as strong as the language of counterfactuals, or why a decision theory that only operates in conditionals will not need human guidance (which uses counterfactuals) to avoid getting counterfactual questions wrong.
Are you under the impression that I think EDT is The Best Thing Ever or something???
EDT gets stuff wrong. It gets transparent newcomb’s wrong. It does not, however kill puppies hoping that this means that there is some sane reason to be killing puppies (because obviously people only ever kill puppies for a good reason).
The reason for this should be obvious [hint: killing puppies in the hope that there’s a good reason does not correlate with having good reasons].
I’ve explained this multiple times. I’m not going to play “person from the non-CDT tribe” any more if you still can’t understand. I don’t even like EDT...
ETA:
Remember when I gave you a mathematical formalism and you ignored it saying “your arrows are the wrong way around” and “CDT is better, by the way”? That’s frustrating. This isn’t a competition.
No, I think you think it’s not as bad as it seems to be. Things not being as they seem is worth investigating, especially about technical topics where additional insight could pay serious dividends.
I apologize for any harshness. I don’t think a tribal lens is useful for this discussion, though; friends can disagree stringently on technical issues. I think that as an epistemic principle it’s worth pursuing such disagreements, even though disagreements in other situations might be a signal of enmity.
It’s not clear to me why you think I ignored your formalism; I needed to understand it to respond how I did.
My view is that there are two options for your formalism:
The formalism should be viewed as a causal graph, and decisions as the do() operator, in which case the decision theory is CDT and the formalism does not capture Newcomb’s problem because Omega is not a perfect predictor. Your formalism describes the situation where you want to be a one-boxer when scanned, and then switch to two-boxing after Omega has left, without Omega knowing that you’ll switch. If this is impossible, you are choosing who to be 5 minutes ago, not your decision now (or your decision now causes who you were 5 minutes ago), and if this is possible, then Omega isn’t a perfect predictor.
The formalism should be viewed as a factorization of a joint probability distribution, and decisions as observations, in which case the decision theory is EDT and the formalism will lead to “managing the news” in other situations. (If your formalism is a causal graph but decisions are observations, then the ‘causal’ part of the graph is unused, and I don’t see a difference between it and a factorization of a joint probability distribution.)
To justify the second view: I do not see a structural difference between
and
whereas with the do() operator, the structural difference is clear (interventions do not propagate backwards across causal arrows). If you want to use language like “the causal connection between “getting out of bed” and “morning” is broken,” that language makes sense with the do() operator but doesn’t make sense with conditioning. How are you supposed to know which correlations are broken, and which aren’t, without using the do() operator?
The difference is that in the first case the diamond graph is still accurate, because our dataset or other evidence that we construct the graph from says that there’s a correlation between 5-minutes-ago and one-box even when the agent doesn’t know the state of agent-5-minutes-ago.
In the second case there should be no connection from
timetoget-out-of-bed, because we haven’t observed the time, and all our samples which would otherwise suggest a correlation involve an agent who has observed the time, and decides whether to get out of bed based on that, so they’re inapplicable to this situation. More strongly, we know that the supposed correlation is mediated by “looking at the clock and deciding to get up if it’s past 8am”, which cannot happen here. There is no correlation until we observe the time.My understanding of this response is “the structural difference between the situations is that the causal graphs are different for the two situations.” I agree with that statement, but I think you have the graphs flipped around; I think the causal graph you drew describes the get-out-of-bed problem, and not Newcomb’s problem. I think the following causal graph fits the “get out of bed?” situation:
Time → I get out of bed → Late, Time → Boss gets out of bed → Late
(I’ve added the second path to make the graphs exactly analogous; suppose I’m only late for work if I arrive after my boss, and if I manage to change what time it is, I’ll change when my boss gets to work because I’ve changed what time it is.)
This has the same form as the Newcomb’s causal graph that you made; only the names have changed. In that situation, you asserted that the direct correlation between “one-box?” and “agent 5 minutes ago” was strong and relevant to our decision, even though we hadn’t observed “agent 5 minutes ago,” and historical agents who played Omega’s game hadn’t observed “agent 5 minutes ago” when they played. I assert that there’s a direct correlation between Time and I-get-out-of-bed which doesn’t depend on observing Time. (The assumption that the correlation between getting out of bed and the time is mediated by looking at the clock is your addition, and doesn’t need to be true; let’s assume the least convenient possible world where it isn’t mediated by that. I can think of a few examples and I’m sure you could as well.)
And so when we calculate P(Late|I-get-out-of-bed) using the method you recommended for Newcomb’s in the last paragraph of this comment, we implicitly marginalize over Time and Boss-gets-out-of-bed, and notice that when we choose to stay in bed, this increases the probability that it’s early, which increases the probability that our boss chooses to stay in bed!
For the two graphs- which only differ in the names of the nodes- to give different mathematical results, we would need to have our math pay attention to the names of the nodes. This is a fatal flaw, because it means we need to either teach our math how to read English, or hand-guide it every time we want to use it.
This automatic understanding of causality is what the mathematical formalism of interventions is good for: while observations flow back up causal chains (seeing my partner get out of bed is evidence about the time, or looking at a clock is evidence about the time), interventions don’t (seeing myself get out of bed isn’t evidence about the time, or adjusting my clock isn’t evidence about the time). All the English and physics surrounding these two problems can be encapsulated in the direction of the causal arrows in the graph, and then the math can proceed automatically.
This requires that different problems have different graphs: the “get out of bed” problem has the graph I describe here, which is identical to your Newcomb’s graph, and the Newcomb’s graph has the arrow pointing from one-box? to agent-5-minutes-ago, rather than the other way around, like I suggested here. With the bed graph I recommend, choosing to stay in bed cannot affect your boss’s arrival time, and with the Newcomb’s graph I recommend, your decision to one-box can affect Omega’s filling of the boxes. We get the sane results- get out of bed and one-box.
This way, all the exoticism of the problem goes into constructing the graph (a graph where causal arrows point backwards in time is exotic!), not into the math of what to do with a graph.
You can turn any ordinary situation into the smoking lesion by postulating mysterious correlations rather than straightforward correlations that work by conscious decisions based on observations. Did you have any realistic examples in mind?
Yes. Suppose that the person always goes to sleep at the same time, and wakes up after a random interval. In the dark of their bedroom (blackout curtains and no clock), they decide whether or not to go back to sleep or get up. Historically, the later in the morning it is, the more likely they are to get up. To make the historical record analogous to Newcomb’s, we might postulate that historically, they have always decided to go back to sleep before 7 AM, and always decided to get up after 7 AM, and the boss’s alarm is set to wake him up at 7 AM. This is not a very realistic postulation, as a stochastic relationship between the two is more realistic, but the parameters of the factorization are not related to the structure of the causal graph (and Newcomb’s isn’t very realistic either).
It’s not obvious to me what you mean by “mysterious correlations” and “straightforward correlations.” Correlations are statistical objects that either exist or don’t, and I don’t know what conscious decision based on observations you’re referring to in the smoking lesion problem. What makes the smoking lesion problem a problem is that the lesions are unobserved.
For example, in an Israeli study, parole decisions are correlated with the time since the parole board last took a break. Is that correlation mysterious, or straightforward? No one familiar with the system (board members, lawyers, etc.) expected the effects the study revealed, but there are plausible explanations for the effect (mental fatigue, low blood sugar, declining mood, all of which are replenished by a meal break).
It might be that by ‘mysterious correlations’ you mean ‘correlations without an obvious underlying causal mechanism’, and by ‘straightforward correlations’ you mean ‘correlations with an obvious underlying causal mechanism.’ It’s not clear to me what the value of that distinction is. Neither joint probability distributions nor causal graphs do not require that the correlations or causal arrows be labeled.
On the meta level, though, I don’t think it’s productive to fight the hypothetical. I strongly recommend reading The Least Convenient Possible World.
Well, the correlations in the smoking lesion problem are mysterious because they aren’t caused by agents observing
lesion|no-lesionand deciding whether to smoke based on that. They are mysterious because it is simply postulated that “the lesion causes smoking without being observed” without any explanation of how, and it is generally assumed that the correlation somehow still applies when you’re deciding what to do using EDT, which I personally have some doubt about (EDT decides what to do based only on preferences and observations, so how can its output be correlated to anything else?).Straightforward correlations are those where, for example, people go out with an umbrella if they see rain clouds forming. The correlation is created by straightforward decision-making based on observations. Simple statistical reasoning suggests that you only have reason to expect these correlations to hold for an EDT agent if the EDT agent makes the same decisions in the same situations. Furthermore, these correlations tend to pose no problem for EDT because the only time an EDT agent is in a position to take an action correlated to some observation in this way (“I observe rain clouds, should I take my umbrella?”), they must have already observed the correlate (“rain clouds”), so EDT makes no attempt to influence it (“whether or not I take my umbrella, I know there are rain clouds already”) .
Returning to the smoking lesion problem, there are a few ways of making the mystery go away. You can suppose that the lesion works by making you smoke even after you (consciously) decide to do something else. In this case the decision of the EDT agent isn’t actually
smoke | don't-smoke, but rather you get to decide a parameter of something else that determines whether you smoke. This makes the lesion not actually a cause of your decision, so you choose-to-smoke, obviously.Alternatively, I was going to analyse the situation where the lesion makes you want to smoke (by altering your decision theory/preferences), but it made my head hurt. I anticipate that EDT wouldn’t smoke in that situation iff you can somehow remain ignorant of your decision or utility function even while implementing EDT, but I can’t be sure.
Basically, the causal reasons behind your data (why do people always get up after 7AM?) matter, because they determine what kind of causal graph you can infer for the situation with an EDT agent with some given set of observations, as opposed to whatever agents are in the dataset.
Postscript regarding LCPW: If I’m trying to argue that EDT doesn’t normally break, then presenting a situation where it does break isn’t necessarily proper LCPW. Because I never argued that it always did the right thing (which would require me to handle edge cases).
No mathematical decision theory requires verbal explanations to be part of the model that it operates on. (It’s true that when learning a causal model from data, you need causal assumptions; but when a problem provides the model rather than the data, this is not necessary.)
You have doubt that this is how EDT, as a mathematical algorithm, operates, or you have some doubt that this is a wise way to construct a decision-making algorithm?
If the second, this is why I think EDT is a subpar decision theory. It sees the world as a joint probability distribution, and does not have the ability to distinguish correlation and causation, which means it cannot know whether or not a correlation applies for any particular action (and so assumes that all do).
If the first, I’m not sure how to clear up your confusion. There is a mindset that programming cultivates, which is that the system does exactly what you tell it to, with the corollary that your intentions have no weight.
The trouble with LCPW is that it’s asymmetric; Eliezer claims that the LCPW is the one where his friend has to face a moral question, and Eliezer’s friend might claim that the LCPW is the one where Eliezer has to face a practical problem.
The way to break the asymmetry is to try to find the most informative comparison. If the hypothetical has been fought, then we learn nothing about morality, because there is no moral problem. If the hypothetical is accepted despite faults, then we learn quite a bit about morality.
The issues with EDT might require ‘edge cases’ to make obvious, but in the same way that the issues with Newtonian dynamics might require ‘edge cases’ to make obvious.
What I’m saying is that the only way to solve any decision theory problem is to learn a causal model from data. It just doesn’t make sense to postulate particular correlations between an EDT agent’s decisions and other things before you even know what EDT decides! The only reason you get away with assuming graphs like
lesion -> (CDT Agent) -> actionfor CDT is because the first thing CDT does when calculating a decision is break all connections to parents by means ofdo(...).Take Jiro’s example. The lesion makes people jump into volcanoes. 100% of them, and no-one else. Furthermore, I’ll postulate that all of them are using decision theory “check if I have the lesion, if so, jump into a volcano, otherwise don’t”. Should you infer the causal graph
lesion -> (EDT decision: jump?) -> diewith a perfect correlation betweenlesionandjump? (Hint: no, that would be stupid, since we’re not using jump-based-on-lesion-decision-theory, we’re using EDT.)In programming, we also say “garbage in, garbage out”. You are feeding EDT garbage input by giving it factually wrong joint probability distributions.
Ok, what about cases where there are multiple causal hypotheses that are observationally indistinguishable:
a → b → c
vs
a ← b ← c
Both models imply the same joint probability distribution p(a,b,c) with a single conditional independence (a independent of c given b) and cannot be told apart without experimentation. That is, you cannot call p(a,b,c) “factually wrong” because the correct causal model implies it. But the wrong causal model implies it too! To figure out which is which requires causal information. You can give it to EDT and it will work—but then it’s not EDT anymore.
I can give you a graph which implies the same independences as my HAART example but has a completely different causal structure, and the procedure you propose here:
http://lesswrong.com/lw/hwq/evidential_decision_theory_selection_bias_and/9d6f
will give the right answer in one case and the wrong answer in another.
The point is, EDT lacks a rich enough input language to avoid getting garbage inputs in lots of standard cases. Or, more precisely, EDT lacks a rich enough input languages to tell when input is garbage and when it isn’t. This is why EDT is a terrible decision theory.
I think there are a couple of confusions this sentence highlights.
First, there are approaches to solving decision theory problems that don’t use causal models. Part of what has made this conversation challenging is that there are several different ways to represent the world- and so even if CDT is the best / natural one, it needs to be distinguished from other approaches. EDT is not CDT in disguise; the two are distinct formulas / approaches.
Second, there are good reasons to modularize the components of the decision theory, so that you can treat learning a model from data separately from making a decision given a model. An algorithm to turn models into decisions should be able to operate on an arbitrary model, where it sees a → b → c as isomorphic to Drunk → Fall → Death.
To tell an anecdote, when my decision analysis professor would teach that subject to petroleum engineers, he quickly learned not to use petroleum examples. Say something like “suppose the probability of striking oil by drilling a well here is 40%” and an engineer’s hand will shoot up, asking “what kind of rock is it?”. The kind of rock is useful for determining whether or not the probability is 40% or something else, but the question totally misses the point of what the professor is trying to teach. The primary example he uses is choosing a location for a party subject to the uncertainty of the weather.
I’m not sure how to interpret this sentence.
The way EDT operates is to perform the following three steps for each possible action in turn:
Assume that I saw myself doing X.
Perform a Bayesian update on this new evidence.
Calculate and record my utility.
It then chooses the possible action which had the highest calculated utility.
One interpretation is you saying that EDT doesn’t make sense, but I’m not sure I agree with what seems to be the stated reason. It looks to me like you’re saying “it doesn’t make sense to assume that you do X until you know what you decide!”, when I think that does make sense, but the problem is using that assumption as Bayesian evidence as if it were an observation.
Ideal Bayesian updates assume logical omniscience, right? Including knowledge about logical fact of what EDT would do for any given input. If you know that you are an EDT agent, and condition on all of your past observations and also on the fact that you do X, but X is not in fact what EDT does given those inputs, then as an ideal Bayesian you will know that you’re conditioning on something impossible. More generally, what update you perform in step 2 depends on EDT’s input-output map, thus making the definition circular.
So, is EDT really underspecified? Or are you supposed to search for a fixed point of the circular definition, if there is one? Or does it use some method other than Bayes for the hypothetical update? Or does an EDT agent really break if it ever finds out its own decision algorithm? Or did I totally misunderstand?
Note that step 1 is “Assume that I saw myself doing X,” not “Assume that EDT outputs X as the optimal action.” I believe that excludes any contradictions along those lines. Does logical omniscience preclude imagining counterfactual worlds?
If I already know “I am EDT”, then “I saw myself doing X” does imply “EDT outputs X as the optimal action”. Logical omniscience doesn’t preclude imagining counterfactual worlds, but imagining counterfactual worlds is a different operation than performing Bayesian updates. CDT constructs counterfactuals by severing some of the edges in its causal graph and then assuming certain values for the nodes that no longer have any causes. TDT does too, except with a different graph and a different choice of edges to sever.
I don’t know how I can fail to communicate so consistently.
Yes, you can technically apply “EDT” to any causal model or (more generally) joint probability distribution containing a “EDT agent decision” node. But in practice this freedom is useless, because to derive an accurate model you generally need to take account of a) the fact that the agent is using EDT and b) any observations the agent does or does not make. To be clear, the input EDT requires is a probabilistic model describing the EDT agent’s situation (not describing historical data of “similar” situations).
There are people here trying to argue against EDT by taking a model describing historical data (such as people following dumb decision theories jumping into volcanoes) and feeding this model directly into EDT. Which is simply wrong. A model that describes the historical behaviour of agents using some other decision theory does not in general accurately describe an EDT agent in the same situation.
The fact that this egregious mistake looks perfectly normal is an artifact of the fact that CDT doesn’t care about causal parents of the “CDT decision” node.
I suspect it’s because what you are referring to as “EDT” is not what experts in the field use that technical term to mean.
nsheppard-EDT is, as far as I can tell, the second half of CDT. Take a causal model and use the do() operator to create the manipulated subgraph that would result taking possible action (as an intervention). Determine the joint probability distribution from the manipulated subgraph. Condition on observing that action with the joint probability distribution, and calculate the probabilistically-weighted mean utility of the possible outcomes. This is isomorphic to CDT, and so referring to it as EDT leads to confusion.
Whatever. I give up.
Here’s a modified version. Instead of a smoking lesion, there’s a “jump into active volcano lesion”. Furthermore, the correlation isn’t as puny as for the smoking lesion. 100% of people with this lesion jump into active volcanoes and die, and nobody else does.
Should you go jump into an active volcano?
Using a decision theory to figure out what decision you should make assumes that you’re capable of making a decision. “The lesion causes you to jump into an active volcano/smoke” and “you can choose whether to jump into an active volcano/smoke” are contradictory. Even “the lesion is correlated (at less than 100%) with jumping into an active volcano/smoking” and “you can choose whether to jump into an active volcano/smoke” are contradictory unless “is correlated with” involves some correlation for people who don’t use decision theory and no correlation for people who do.
Agreed.
Doesn’t this seem sort of realistic, actually? Decisions made with System 1 and System 2, to use Kahneman’s language, might have entirely different underlying algorithms. (There is some philosophical trouble about how far we can push the idea of an ‘intervention’, but I think for human-scale decisions there is a meaningful difference between interventions and observations such that CDT distinguishing between them is a feature.)
This maps onto an objection by proponents of EDT that the observational data might not be from people using EDT, and thus the correlation may disappear when EDT comes onto the stage. I think that objection proves too much- suppose all of our observational data on the health effects of jumping off cliffs comes from subjects who were not using EDT (suppose they were drunk). I don’t see a reason inside the decision theory for differentiating between the effects of EDT on the correlation between jumping off the cliff and the effects of EDT on the correlation between smoking and having the lesion.
These two situations correspond to two different causal structures—Drunk → Fall → Death and Smoke ← Lesion → Cancer—which could have the same joint probability distribution. The directionality of the arrow is something that CDT can make use of to tell that the two situations will respond differently to interventions at Drunk and Smoke: it is dangerous to be drunk around cliffs, but not to smoke (in this hypothetical world).
EDT cannot make use of those arrows. It just has Drunk—Fall—Death and Smoke—Lesion—Cancer (where it knows that the correlations between Drunk and Death are mediated by Fall, and the correlations between Smoke and Cancer are mediated by Lesion). If we suppose that adding an EDT node might mean that the correlation between Smoke and Lesion (and thus Cancer) might be mediated by EDT, then we must also suppose that adding an EDT node might mean that the correlation between Drunk and Fall (and thus Death) might be mediated by EDT.
(I should point out that the EDT node describes whether or not EDT was used to decide to drink, not to decide whether or not to fall off the cliff, by analogy of using EDT to decide whether or not to smoke, rather than deciding whether or not to have a lesion.)
there’s another CRUCIAL difference regarding the Newcombs problem: there’s always a chance you’re in a simulation being run by Omega. I think if you can account for that, it SHOULD patch most decent decision-theories up. I’m willing to be quite flexible in my understanding of which theories get patched up or not.
this has the BIG advantage of NOT requiring non-linear causality in the model-it just gives a flow from simulation->”real”world.
Yes, reflective consistency tends to make things better.
um...that wasn’t sarcastic, was it? I just ran low on mental energy so...
anyways, the downside is you have to figure out how to dissolve all or most of the anthropic paradoxes when evaluating simulation chance.