I’m not actually advocating EDT. After all, it two-boxes on transparent Newcomb’s, which is a clear mistake. I’m just trying to explain how it’s not as bad as it seems to be.
I don’t see why you think that EDT isn’t “as bad as it seems to be,” yet. I see lots of verbal explanations of why EDT wouldn’t do X, but what I’m looking for is a mathematical reason why you think the language of conditionals is as strong as the language of counterfactuals, or why a decision theory that only operates in conditionals will not need human guidance (which uses counterfactuals) to avoid getting counterfactual questions wrong.
Are you under the impression that I think EDT is The Best Thing Ever or something???
EDT gets stuff wrong. It gets transparent newcomb’s wrong. It does not, however kill puppies hoping that this means that there is some sane reason to be killing puppies (because obviously people only ever kill puppies for a good reason).
The reason for this should be obvious [hint: killing puppies in the hope that there’s a good reason does not correlate with having good reasons].
I’ve explained this multiple times. I’m not going to play “person from the non-CDT tribe” any more if you still can’t understand. I don’t even like EDT...
ETA:
mathematical reason
Remember when I gave you a mathematical formalism and you ignored it saying “your arrows are the wrong way around” and “CDT is better, by the way”? That’s frustrating. This isn’t a competition.
Are you under the impression that I think EDT is The Best Thing Ever or something???
No, I think you think it’s not as bad as it seems to be. Things not being as they seem is worth investigating, especially about technical topics where additional insight could pay serious dividends.
I’m not going to play “person from the non-CDT tribe” any more if you still can’t understand.
I apologize for any harshness. I don’t think a tribal lens is useful for this discussion, though; friends can disagree stringently on technical issues. I think that as an epistemic principle it’s worth pursuing such disagreements, even though disagreements in other situations might be a signal of enmity.
Remember when I gave you a mathematical formalism and you ignored it saying “your arrows are the wrong way around” and “CDT is better, by the way”?
It’s not clear to me why you think I ignored your formalism; I needed to understand it to respond how I did.
My view is that there are two options for your formalism:
The formalism should be viewed as a causal graph, and decisions as the do() operator, in which case the decision theory is CDT and the formalism does not capture Newcomb’s problem because Omega is not a perfect predictor. Your formalism describes the situation where you want to be a one-boxer when scanned, and then switch to two-boxing after Omega has left, without Omega knowing that you’ll switch. If this is impossible, you are choosing who to be 5 minutes ago, not your decision now (or your decision now causes who you were 5 minutes ago), and if this is possible, then Omega isn’t a perfect predictor.
The formalism should be viewed as a factorization of a joint probability distribution, and decisions as observations, in which case the decision theory is EDT and the formalism will lead to “managing the news” in other situations. (If your formalism is a causal graph but decisions are observations, then the ‘causal’ part of the graph is unused, and I don’t see a difference between it and a factorization of a joint probability distribution.)
To justify the second view: I do not see a structural difference between
We don’t know whether agent-5-minutes-ago was the sort that would make omega fill both boxes or not (so it’s not an observation), but we do know that there’s a direct correlation between that and our one-boxing.
and
We don’t know what time it is (so it’s not an observation), but we do know that there’s a direct correlation between that and our getting out of bed.
whereas with the do() operator, the structural difference is clear (interventions do not propagate backwards across causal arrows). If you want to use language like “the causal connection between “getting out of bed” and “morning” is broken,” that language makes sense with the do() operator but doesn’t make sense with conditioning. How are you supposed to know which correlations are broken, and which aren’t, without using the do() operator?
To justify the second view: I do not see a structural difference between
We don’t know whether agent-5-minutes-ago was the sort that would make omega fill both boxes or not (so it’s not an observation), but we do know that there’s a direct correlation between that and our one-boxing.
and
We don’t know what time it is (so it’s not an observation), but we do know that there’s a direct correlation between that and our getting out of bed.
The difference is that in the first case the diamond graph is still accurate, because our dataset or other evidence that we construct the graph from says that there’s a correlation between 5-minutes-ago and one-box even when the agent doesn’t know the state of agent-5-minutes-ago.
In the second case there should be no connection from time to get-out-of-bed, because we haven’t observed the time, and all our samples which would otherwise suggest a correlation involve an agent who has observed the time, and decides whether to get out of bed based on that, so they’re inapplicable to this situation. More strongly, we know that the supposed correlation is mediated by “looking at the clock and deciding to get up if it’s past 8am”, which cannot happen here. There is no correlation until we observe the time.
The difference is that in the first case the diamond graph is still accurate
My understanding of this response is “the structural difference between the situations is that the causal graphs are different for the two situations.” I agree with that statement, but I think you have the graphs flipped around; I think the causal graph you drew describes the get-out-of-bed problem, and not Newcomb’s problem. I think the following causal graph fits the “get out of bed?” situation:
Time → I get out of bed → Late, Time → Boss gets out of bed → Late
(I’ve added the second path to make the graphs exactly analogous; suppose I’m only late for work if I arrive after my boss, and if I manage to change what time it is, I’ll change when my boss gets to work because I’ve changed what time it is.)
This has the same form as the Newcomb’s causal graph that you made; only the names have changed. In that situation, you asserted that the direct correlation between “one-box?” and “agent 5 minutes ago” was strong and relevant to our decision, even though we hadn’t observed “agent 5 minutes ago,” and historical agents who played Omega’s game hadn’t observed “agent 5 minutes ago” when they played. I assert that there’s a direct correlation between Time and I-get-out-of-bed which doesn’t depend on observing Time. (The assumption that the correlation between getting out of bed and the time is mediated by looking at the clock is your addition, and doesn’t need to be true; let’s assume the least convenient possible world where it isn’t mediated by that. I can think of a few examples and I’m sure you could as well.)
And so when we calculate P(Late|I-get-out-of-bed) using the method you recommended for Newcomb’s in the last paragraph of this comment, we implicitly marginalize over Time and Boss-gets-out-of-bed, and notice that when we choose to stay in bed, this increases the probability that it’s early, which increases the probability that our boss chooses to stay in bed!
For the two graphs- which only differ in the names of the nodes- to give different mathematical results, we would need to have our math pay attention to the names of the nodes. This is a fatal flaw, because it means we need to either teach our math how to read English, or hand-guide it every time we want to use it.
This automatic understanding of causality is what the mathematical formalism of interventions is good for: while observations flow back up causal chains (seeing my partner get out of bed is evidence about the time, or looking at a clock is evidence about the time), interventions don’t (seeing myself get out of bed isn’t evidence about the time, or adjusting my clock isn’t evidence about the time). All the English and physics surrounding these two problems can be encapsulated in the direction of the causal arrows in the graph, and then the math can proceed automatically.
This requires that different problems have different graphs: the “get out of bed” problem has the graph I describe here, which is identical to your Newcomb’s graph, and the Newcomb’s graph has the arrow pointing from one-box? to agent-5-minutes-ago, rather than the other way around, like I suggested here. With the bed graph I recommend, choosing to stay in bed cannot affect your boss’s arrival time, and with the Newcomb’s graph I recommend, your decision to one-box can affect Omega’s filling of the boxes. We get the sane results- get out of bed and one-box.
This way, all the exoticism of the problem goes into constructing the graph (a graph where causal arrows point backwards in time is exotic!), not into the math of what to do with a graph.
I assert that there’s a direct correlation between Time and I-get-out-of-bed which doesn’t depend on observing Time. (The assumption that the correlation between getting out of bed and the time is mediated by looking at the clock is your addition, and doesn’t need to be true; let’s assume the least convenient possible world where it isn’t mediated by that. I can think of a few examples and I’m sure you could as well.)
You can turn any ordinary situation into the smoking lesion by postulating mysterious correlations rather than straightforward correlations that work by conscious decisions based on observations. Did you have any realistic examples in mind?
Yes. Suppose that the person always goes to sleep at the same time, and wakes up after a random interval. In the dark of their bedroom (blackout curtains and no clock), they decide whether or not to go back to sleep or get up. Historically, the later in the morning it is, the more likely they are to get up. To make the historical record analogous to Newcomb’s, we might postulate that historically, they have always decided to go back to sleep before 7 AM, and always decided to get up after 7 AM, and the boss’s alarm is set to wake him up at 7 AM. This is not a very realistic postulation, as a stochastic relationship between the two is more realistic, but the parameters of the factorization are not related to the structure of the causal graph (and Newcomb’s isn’t very realistic either).
You can turn any ordinary situation into the smoking lesion by postulating mysterious correlations rather than straightforward correlations that work by conscious decisions based on observations.
It’s not obvious to me what you mean by “mysterious correlations” and “straightforward correlations.” Correlations are statistical objects that either exist or don’t, and I don’t know what conscious decision based on observations you’re referring to in the smoking lesion problem. What makes the smoking lesion problem a problem is that the lesions are unobserved.
For example, in an Israeli study, parole decisions are correlated with the time since the parole board last took a break. Is that correlation mysterious, or straightforward? No one familiar with the system (board members, lawyers, etc.) expected the effects the study revealed, but there are plausible explanations for the effect (mental fatigue, low blood sugar, declining mood, all of which are replenished by a meal break).
It might be that by ‘mysterious correlations’ you mean ‘correlations without an obvious underlying causal mechanism’, and by ‘straightforward correlations’ you mean ‘correlations with an obvious underlying causal mechanism.’ It’s not clear to me what the value of that distinction is. Neither joint probability distributions nor causal graphs do not require that the correlations or causal arrows be labeled.
On the meta level, though, I don’t think it’s productive to fight the hypothetical. I strongly recommend reading The Least Convenient Possible World.
It’s not obvious to me what you mean by “mysterious correlations” and “straightforward correlations.” Correlations are statistical objects that either exist or don’t, and I don’t know what conscious decision based on observations you’re referring to in the smoking lesion problem. What makes the smoking lesion problem a problem is that the lesions are unobserved.
Well, the correlations in the smoking lesion problem are mysterious because they aren’t caused by agents observing lesion|no-lesion and deciding whether to smoke based on that. They are mysterious because it is simply postulated that “the lesion causes smoking without being observed” without any explanation of how, and it is generally assumed that the correlation somehow still applies when you’re deciding what to do using EDT, which I personally have some doubt about (EDT decides what to do based only on preferences and observations, so how can its output be correlated to anything else?).
Straightforward correlations are those where, for example, people go out with an umbrella if they see rain clouds forming. The correlation is created by straightforward decision-making based on observations. Simple statistical reasoning suggests that you only have reason to expect these correlations to hold for an EDT agent if the EDT agent makes the same decisions in the same situations. Furthermore, these correlations tend to pose no problem for EDT because the only time an EDT agent is in a position to take an action correlated to some observation in this way (“I observe rain clouds, should I take my umbrella?”), they must have already observed the correlate (“rain clouds”), so EDT makes no attempt to influence it (“whether or not I take my umbrella, I know there are rain clouds already”) .
Returning to the smoking lesion problem, there are a few ways of making the mystery go away. You can suppose that the lesion works by making you smoke even after you (consciously) decide to do something else. In this case the decision of the EDT agent isn’t actually smoke | don't-smoke, but rather you get to decide a parameter of something else that determines whether you smoke. This makes the lesion not actually a cause of your decision, so you choose-to-smoke, obviously.
Alternatively, I was going to analyse the situation where the lesion makes you want to smoke (by altering your decision theory/preferences), but it made my head hurt. I anticipate that EDT wouldn’t smoke in that situation iff you can somehow remain ignorant of your decision or utility function even while implementing EDT, but I can’t be sure.
Basically, the causal reasons behind your data (why do people always get up after 7AM?) matter, because they determine what kind of causal graph you can infer for the situation with an EDT agent with some given set of observations, as opposed to whatever agents are in the dataset.
Postscript regarding LCPW: If I’m trying to argue that EDT doesn’t normally break, then presenting a situation where it does break isn’t necessarily proper LCPW. Because I never argued that it always did the right thing (which would require me to handle edge cases).
it is simply postulated that “the lesion causes smoking without being observed” without any explanation of how
No mathematical decision theory requires verbal explanations to be part of the model that it operates on. (It’s true that when learning a causal model from data, you need causal assumptions; but when a problem provides the model rather than the data, this is not necessary.)
it is generally assumed that the correlation somehow still applies when you’re deciding what to do using EDT, which I personally have some doubt about
You have doubt that this is how EDT, as a mathematical algorithm, operates, or you have some doubt that this is a wise way to construct a decision-making algorithm?
If the second, this is why I think EDT is a subpar decision theory. It sees the world as a joint probability distribution, and does not have the ability to distinguish correlation and causation, which means it cannot know whether or not a correlation applies for any particular action (and so assumes that all do).
If the first, I’m not sure how to clear up your confusion. There is a mindset that programming cultivates, which is that the system does exactly what you tell it to, with the corollary that your intentions have no weight.
If I’m trying to argue that EDT doesn’t normally break, then presenting a situation where it does break isn’t necessarily proper LCPW.
The trouble with LCPW is that it’s asymmetric; Eliezer claims that the LCPW is the one where his friend has to face a moral question, and Eliezer’s friend might claim that the LCPW is the one where Eliezer has to face a practical problem.
The way to break the asymmetry is to try to find the most informative comparison. If the hypothetical has been fought, then we learn nothing about morality, because there is no moral problem. If the hypothetical is accepted despite faults, then we learn quite a bit about morality.
The issues with EDT might require ‘edge cases’ to make obvious, but in the same way that the issues with Newtonian dynamics might require ‘edge cases’ to make obvious.
No mathematical decision theory requires verbal explanations to be part of the model that it operates on. (It’s true that when learning a causal model from data, you need causal assumptions; but when a problem provides the model rather than the data, this is not necessary.)
What I’m saying is that the only way to solve any decision theory problem is to learn a causal model from data. It just doesn’t make sense to postulate particular correlations between an EDT agent’s decisions and other things before you even know what EDT decides! The only reason you get away with assuming graphs like lesion -> (CDT Agent) -> action for CDT is because the first thing CDT does when calculating a decision is break all connections to parents by means of do(...).
Take Jiro’s example. The lesion makes people jump into volcanoes. 100% of them, and no-one else. Furthermore, I’ll postulate that all of them are using decision theory “check if I have the lesion, if so, jump into a volcano, otherwise don’t”. Should you infer the causal graph lesion -> (EDT decision: jump?) -> die with a perfect correlation between lesion and jump? (Hint: no, that would be stupid, since we’re not using jump-based-on-lesion-decision-theory, we’re using EDT.)
There is a mindset that programming cultivates, which is that the system does exactly what you tell it to, with the corollary that your intentions have no weight.
In programming, we also say “garbage in, garbage out”. You are feeding EDT garbage input by giving it factually wrong joint probability distributions.
Ok, what about cases where there are multiple causal hypotheses that are observationally indistinguishable:
a → b → c
vs
a ← b ← c
Both models imply the same joint probability distribution p(a,b,c) with a single conditional independence (a independent of c given b) and cannot be told apart without experimentation. That is, you cannot call p(a,b,c) “factually wrong” because the correct causal model implies it. But the wrong causal model implies it too! To figure out which is which requires causal information. You can give it to EDT and it will work—but then it’s not EDT anymore.
I can give you a graph which implies the same independences as my HAART example but has a completely different causal structure, and the procedure you propose here:
will give the right answer in one case and the wrong answer in another.
The point is, EDT lacks a rich enough input language to avoid getting garbage inputs in lots of standard cases. Or, more precisely, EDT lacks a rich enough input languages to tell when input is garbage and when it isn’t. This is why EDT is a terrible decision theory.
What I’m saying is that the only way to solve any decision theory problem is to learn a causal model from data.
I think there are a couple of confusions this sentence highlights.
First, there are approaches to solving decision theory problems that don’t use causal models. Part of what has made this conversation challenging is that there are several different ways to represent the world- and so even if CDT is the best / natural one, it needs to be distinguished from other approaches. EDT is not CDT in disguise; the two are distinct formulas / approaches.
Second, there are good reasons to modularize the components of the decision theory, so that you can treat learning a model from data separately from making a decision given a model. An algorithm to turn models into decisions should be able to operate on an arbitrary model, where it sees a → b → c as isomorphic to Drunk → Fall → Death.
To tell an anecdote, when my decision analysis professor would teach that subject to petroleum engineers, he quickly learned not to use petroleum examples. Say something like “suppose the probability of striking oil by drilling a well here is 40%” and an engineer’s hand will shoot up, asking “what kind of rock is it?”. The kind of rock is useful for determining whether or not the probability is 40% or something else, but the question totally misses the point of what the professor is trying to teach. The primary example he uses is choosing a location for a party subject to the uncertainty of the weather.
It just doesn’t make sense to postulate particular correlations between an EDT agent’s decisions and other things before you even know what EDT decides!
I’m not sure how to interpret this sentence.
The way EDT operates is to perform the following three steps for each possible action in turn:
Assume that I saw myself doing X.
Perform a Bayesian update on this new evidence.
Calculate and record my utility.
It then chooses the possible action which had the highest calculated utility.
One interpretation is you saying that EDT doesn’t make sense, but I’m not sure I agree with what seems to be the stated reason. It looks to me like you’re saying “it doesn’t make sense to assume that you do X until you know what you decide!”, when I think that does make sense, but the problem is using that assumption as Bayesian evidence as if it were an observation.
The way EDT operates is to perform the following three steps for each possible action in turn:
Assume that I saw myself doing X.
Perform a Bayesian update on this new evidence.
Calculate and record my utility.
Ideal Bayesian updates assume logical omniscience, right? Including knowledge about logical fact of what EDT would do for any given input. If you know that you are an EDT agent, and condition on all of your past observations and also on the fact that you do X, but X is not in fact what EDT does given those inputs, then as an ideal Bayesian you will know that you’re conditioning on something impossible. More generally, what update you perform in step 2 depends on EDT’s input-output map, thus making the definition circular.
So, is EDT really underspecified? Or are you supposed to search for a fixed point of the circular definition, if there is one? Or does it use some method other than Bayes for the hypothetical update? Or does an EDT agent really break if it ever finds out its own decision algorithm? Or did I totally misunderstand?
Ideal Bayesian updates assume logical omniscience, right? Including knowledge about logical fact of what EDT would do for any given input.
Note that step 1 is “Assume that I saw myself doing X,” not “Assume that EDT outputs X as the optimal action.” I believe that excludes any contradictions along those lines. Does logical omniscience preclude imagining counterfactual worlds?
If I already know “I am EDT”, then “I saw myself doing X” does imply “EDT outputs X as the optimal action”. Logical omniscience doesn’t preclude imagining counterfactual worlds, but imagining counterfactual worlds is a different operation than performing Bayesian updates. CDT constructs counterfactuals by severing some of the edges in its causal graph and then assuming certain values for the nodes that no longer have any causes. TDT does too, except with a different graph and a different choice of edges to sever.
I don’t know how I can fail to communicate so consistently.
Yes, you can technically apply “EDT” to any causal model or (more generally) joint probability distribution containing a “EDT agent decision” node. But in practice this freedom is useless, because to derive an accurate model you generally need to take account of a) the fact that the agent is using EDT and b) any observations the agent does or does not make. To be clear, the input EDT requires is a probabilistic model describing the EDT agent’s situation (not describing historical data of “similar” situations).
There are people here trying to argue against EDT by taking a model describing historical data (such as people following dumb decision theories jumping into volcanoes) and feeding this model directly into EDT. Which is simply wrong. A model that describes the historical behaviour of agents using some other decision theory does not in general accurately describe an EDT agent in the same situation.
The fact that this egregious mistake looks perfectly normal is an artifact of the fact that CDT doesn’t care about causal parents of the “CDT decision” node.
I don’t know how I can fail to communicate so consistently.
I suspect it’s because what you are referring to as “EDT” is not what experts in the field use that technical term to mean.
nsheppard-EDT is, as far as I can tell, the second half of CDT. Take a causal model and use the do() operator to create the manipulated subgraph that would result taking possible action (as an intervention). Determine the joint probability distribution from the manipulated subgraph. Condition on observing that action with the joint probability distribution, and calculate the probabilistically-weighted mean utility of the possible outcomes. This is isomorphic to CDT, and so referring to it as EDT leads to confusion.
Here’s a modified version. Instead of a smoking lesion, there’s a “jump into active volcano lesion”. Furthermore, the correlation isn’t as puny as for the smoking lesion. 100% of people with this lesion jump into active volcanoes and die, and nobody else does.
Should you go jump into an active volcano?
Using a decision theory to figure out what decision you should make assumes that you’re capable of making a decision. “The lesion causes you to jump into an active volcano/smoke” and “you can choose whether to jump into an active volcano/smoke” are contradictory. Even “the lesion is correlated (at less than 100%) with jumping into an active volcano/smoking” and “you can choose whether to jump into an active volcano/smoke” are contradictory unless “is correlated with” involves some correlation for people who don’t use decision theory and no correlation for people who do.
Using a decision theory to figure out what decision you should make assumes that you’re capable of making a decision.
Agreed.
unless “is correlated with” involves some correlation for people who don’t use decision theory and no correlation for people who do.
Doesn’t this seem sort of realistic, actually? Decisions made with System 1 and System 2, to use Kahneman’s language, might have entirely different underlying algorithms. (There is some philosophical trouble about how far we can push the idea of an ‘intervention’, but I think for human-scale decisions there is a meaningful difference between interventions and observations such that CDT distinguishing between them is a feature.)
This maps onto an objection by proponents of EDT that the observational data might not be from people using EDT, and thus the correlation may disappear when EDT comes onto the stage. I think that objection proves too much- suppose all of our observational data on the health effects of jumping off cliffs comes from subjects who were not using EDT (suppose they were drunk). I don’t see a reason inside the decision theory for differentiating between the effects of EDT on the correlation between jumping off the cliff and the effects of EDT on the correlation between smoking and having the lesion.
These two situations correspond to two different causal structures—Drunk → Fall → Death and Smoke ← Lesion → Cancer—which could have the same joint probability distribution. The directionality of the arrow is something that CDT can make use of to tell that the two situations will respond differently to interventions at Drunk and Smoke: it is dangerous to be drunk around cliffs, but not to smoke (in this hypothetical world).
EDT cannot make use of those arrows. It just has Drunk—Fall—Death and Smoke—Lesion—Cancer (where it knows that the correlations between Drunk and Death are mediated by Fall, and the correlations between Smoke and Cancer are mediated by Lesion). If we suppose that adding an EDT node might mean that the correlation between Smoke and Lesion (and thus Cancer) might be mediated by EDT, then we must also suppose that adding an EDT node might mean that the correlation between Drunk and Fall (and thus Death) might be mediated by EDT.
(I should point out that the EDT node describes whether or not EDT was used to decide to drink, not to decide whether or not to fall off the cliff, by analogy of using EDT to decide whether or not to smoke, rather than deciding whether or not to have a lesion.)
I don’t see why you think that EDT isn’t “as bad as it seems to be,” yet. I see lots of verbal explanations of why EDT wouldn’t do X, but what I’m looking for is a mathematical reason why you think the language of conditionals is as strong as the language of counterfactuals, or why a decision theory that only operates in conditionals will not need human guidance (which uses counterfactuals) to avoid getting counterfactual questions wrong.
Are you under the impression that I think EDT is The Best Thing Ever or something???
EDT gets stuff wrong. It gets transparent newcomb’s wrong. It does not, however kill puppies hoping that this means that there is some sane reason to be killing puppies (because obviously people only ever kill puppies for a good reason).
The reason for this should be obvious [hint: killing puppies in the hope that there’s a good reason does not correlate with having good reasons].
I’ve explained this multiple times. I’m not going to play “person from the non-CDT tribe” any more if you still can’t understand. I don’t even like EDT...
ETA:
Remember when I gave you a mathematical formalism and you ignored it saying “your arrows are the wrong way around” and “CDT is better, by the way”? That’s frustrating. This isn’t a competition.
No, I think you think it’s not as bad as it seems to be. Things not being as they seem is worth investigating, especially about technical topics where additional insight could pay serious dividends.
I apologize for any harshness. I don’t think a tribal lens is useful for this discussion, though; friends can disagree stringently on technical issues. I think that as an epistemic principle it’s worth pursuing such disagreements, even though disagreements in other situations might be a signal of enmity.
It’s not clear to me why you think I ignored your formalism; I needed to understand it to respond how I did.
My view is that there are two options for your formalism:
The formalism should be viewed as a causal graph, and decisions as the do() operator, in which case the decision theory is CDT and the formalism does not capture Newcomb’s problem because Omega is not a perfect predictor. Your formalism describes the situation where you want to be a one-boxer when scanned, and then switch to two-boxing after Omega has left, without Omega knowing that you’ll switch. If this is impossible, you are choosing who to be 5 minutes ago, not your decision now (or your decision now causes who you were 5 minutes ago), and if this is possible, then Omega isn’t a perfect predictor.
The formalism should be viewed as a factorization of a joint probability distribution, and decisions as observations, in which case the decision theory is EDT and the formalism will lead to “managing the news” in other situations. (If your formalism is a causal graph but decisions are observations, then the ‘causal’ part of the graph is unused, and I don’t see a difference between it and a factorization of a joint probability distribution.)
To justify the second view: I do not see a structural difference between
and
whereas with the do() operator, the structural difference is clear (interventions do not propagate backwards across causal arrows). If you want to use language like “the causal connection between “getting out of bed” and “morning” is broken,” that language makes sense with the do() operator but doesn’t make sense with conditioning. How are you supposed to know which correlations are broken, and which aren’t, without using the do() operator?
The difference is that in the first case the diamond graph is still accurate, because our dataset or other evidence that we construct the graph from says that there’s a correlation between 5-minutes-ago and one-box even when the agent doesn’t know the state of agent-5-minutes-ago.
In the second case there should be no connection from
timetoget-out-of-bed, because we haven’t observed the time, and all our samples which would otherwise suggest a correlation involve an agent who has observed the time, and decides whether to get out of bed based on that, so they’re inapplicable to this situation. More strongly, we know that the supposed correlation is mediated by “looking at the clock and deciding to get up if it’s past 8am”, which cannot happen here. There is no correlation until we observe the time.My understanding of this response is “the structural difference between the situations is that the causal graphs are different for the two situations.” I agree with that statement, but I think you have the graphs flipped around; I think the causal graph you drew describes the get-out-of-bed problem, and not Newcomb’s problem. I think the following causal graph fits the “get out of bed?” situation:
Time → I get out of bed → Late, Time → Boss gets out of bed → Late
(I’ve added the second path to make the graphs exactly analogous; suppose I’m only late for work if I arrive after my boss, and if I manage to change what time it is, I’ll change when my boss gets to work because I’ve changed what time it is.)
This has the same form as the Newcomb’s causal graph that you made; only the names have changed. In that situation, you asserted that the direct correlation between “one-box?” and “agent 5 minutes ago” was strong and relevant to our decision, even though we hadn’t observed “agent 5 minutes ago,” and historical agents who played Omega’s game hadn’t observed “agent 5 minutes ago” when they played. I assert that there’s a direct correlation between Time and I-get-out-of-bed which doesn’t depend on observing Time. (The assumption that the correlation between getting out of bed and the time is mediated by looking at the clock is your addition, and doesn’t need to be true; let’s assume the least convenient possible world where it isn’t mediated by that. I can think of a few examples and I’m sure you could as well.)
And so when we calculate P(Late|I-get-out-of-bed) using the method you recommended for Newcomb’s in the last paragraph of this comment, we implicitly marginalize over Time and Boss-gets-out-of-bed, and notice that when we choose to stay in bed, this increases the probability that it’s early, which increases the probability that our boss chooses to stay in bed!
For the two graphs- which only differ in the names of the nodes- to give different mathematical results, we would need to have our math pay attention to the names of the nodes. This is a fatal flaw, because it means we need to either teach our math how to read English, or hand-guide it every time we want to use it.
This automatic understanding of causality is what the mathematical formalism of interventions is good for: while observations flow back up causal chains (seeing my partner get out of bed is evidence about the time, or looking at a clock is evidence about the time), interventions don’t (seeing myself get out of bed isn’t evidence about the time, or adjusting my clock isn’t evidence about the time). All the English and physics surrounding these two problems can be encapsulated in the direction of the causal arrows in the graph, and then the math can proceed automatically.
This requires that different problems have different graphs: the “get out of bed” problem has the graph I describe here, which is identical to your Newcomb’s graph, and the Newcomb’s graph has the arrow pointing from one-box? to agent-5-minutes-ago, rather than the other way around, like I suggested here. With the bed graph I recommend, choosing to stay in bed cannot affect your boss’s arrival time, and with the Newcomb’s graph I recommend, your decision to one-box can affect Omega’s filling of the boxes. We get the sane results- get out of bed and one-box.
This way, all the exoticism of the problem goes into constructing the graph (a graph where causal arrows point backwards in time is exotic!), not into the math of what to do with a graph.
You can turn any ordinary situation into the smoking lesion by postulating mysterious correlations rather than straightforward correlations that work by conscious decisions based on observations. Did you have any realistic examples in mind?
Yes. Suppose that the person always goes to sleep at the same time, and wakes up after a random interval. In the dark of their bedroom (blackout curtains and no clock), they decide whether or not to go back to sleep or get up. Historically, the later in the morning it is, the more likely they are to get up. To make the historical record analogous to Newcomb’s, we might postulate that historically, they have always decided to go back to sleep before 7 AM, and always decided to get up after 7 AM, and the boss’s alarm is set to wake him up at 7 AM. This is not a very realistic postulation, as a stochastic relationship between the two is more realistic, but the parameters of the factorization are not related to the structure of the causal graph (and Newcomb’s isn’t very realistic either).
It’s not obvious to me what you mean by “mysterious correlations” and “straightforward correlations.” Correlations are statistical objects that either exist or don’t, and I don’t know what conscious decision based on observations you’re referring to in the smoking lesion problem. What makes the smoking lesion problem a problem is that the lesions are unobserved.
For example, in an Israeli study, parole decisions are correlated with the time since the parole board last took a break. Is that correlation mysterious, or straightforward? No one familiar with the system (board members, lawyers, etc.) expected the effects the study revealed, but there are plausible explanations for the effect (mental fatigue, low blood sugar, declining mood, all of which are replenished by a meal break).
It might be that by ‘mysterious correlations’ you mean ‘correlations without an obvious underlying causal mechanism’, and by ‘straightforward correlations’ you mean ‘correlations with an obvious underlying causal mechanism.’ It’s not clear to me what the value of that distinction is. Neither joint probability distributions nor causal graphs do not require that the correlations or causal arrows be labeled.
On the meta level, though, I don’t think it’s productive to fight the hypothetical. I strongly recommend reading The Least Convenient Possible World.
Well, the correlations in the smoking lesion problem are mysterious because they aren’t caused by agents observing
lesion|no-lesionand deciding whether to smoke based on that. They are mysterious because it is simply postulated that “the lesion causes smoking without being observed” without any explanation of how, and it is generally assumed that the correlation somehow still applies when you’re deciding what to do using EDT, which I personally have some doubt about (EDT decides what to do based only on preferences and observations, so how can its output be correlated to anything else?).Straightforward correlations are those where, for example, people go out with an umbrella if they see rain clouds forming. The correlation is created by straightforward decision-making based on observations. Simple statistical reasoning suggests that you only have reason to expect these correlations to hold for an EDT agent if the EDT agent makes the same decisions in the same situations. Furthermore, these correlations tend to pose no problem for EDT because the only time an EDT agent is in a position to take an action correlated to some observation in this way (“I observe rain clouds, should I take my umbrella?”), they must have already observed the correlate (“rain clouds”), so EDT makes no attempt to influence it (“whether or not I take my umbrella, I know there are rain clouds already”) .
Returning to the smoking lesion problem, there are a few ways of making the mystery go away. You can suppose that the lesion works by making you smoke even after you (consciously) decide to do something else. In this case the decision of the EDT agent isn’t actually
smoke | don't-smoke, but rather you get to decide a parameter of something else that determines whether you smoke. This makes the lesion not actually a cause of your decision, so you choose-to-smoke, obviously.Alternatively, I was going to analyse the situation where the lesion makes you want to smoke (by altering your decision theory/preferences), but it made my head hurt. I anticipate that EDT wouldn’t smoke in that situation iff you can somehow remain ignorant of your decision or utility function even while implementing EDT, but I can’t be sure.
Basically, the causal reasons behind your data (why do people always get up after 7AM?) matter, because they determine what kind of causal graph you can infer for the situation with an EDT agent with some given set of observations, as opposed to whatever agents are in the dataset.
Postscript regarding LCPW: If I’m trying to argue that EDT doesn’t normally break, then presenting a situation where it does break isn’t necessarily proper LCPW. Because I never argued that it always did the right thing (which would require me to handle edge cases).
No mathematical decision theory requires verbal explanations to be part of the model that it operates on. (It’s true that when learning a causal model from data, you need causal assumptions; but when a problem provides the model rather than the data, this is not necessary.)
You have doubt that this is how EDT, as a mathematical algorithm, operates, or you have some doubt that this is a wise way to construct a decision-making algorithm?
If the second, this is why I think EDT is a subpar decision theory. It sees the world as a joint probability distribution, and does not have the ability to distinguish correlation and causation, which means it cannot know whether or not a correlation applies for any particular action (and so assumes that all do).
If the first, I’m not sure how to clear up your confusion. There is a mindset that programming cultivates, which is that the system does exactly what you tell it to, with the corollary that your intentions have no weight.
The trouble with LCPW is that it’s asymmetric; Eliezer claims that the LCPW is the one where his friend has to face a moral question, and Eliezer’s friend might claim that the LCPW is the one where Eliezer has to face a practical problem.
The way to break the asymmetry is to try to find the most informative comparison. If the hypothetical has been fought, then we learn nothing about morality, because there is no moral problem. If the hypothetical is accepted despite faults, then we learn quite a bit about morality.
The issues with EDT might require ‘edge cases’ to make obvious, but in the same way that the issues with Newtonian dynamics might require ‘edge cases’ to make obvious.
What I’m saying is that the only way to solve any decision theory problem is to learn a causal model from data. It just doesn’t make sense to postulate particular correlations between an EDT agent’s decisions and other things before you even know what EDT decides! The only reason you get away with assuming graphs like
lesion -> (CDT Agent) -> actionfor CDT is because the first thing CDT does when calculating a decision is break all connections to parents by means ofdo(...).Take Jiro’s example. The lesion makes people jump into volcanoes. 100% of them, and no-one else. Furthermore, I’ll postulate that all of them are using decision theory “check if I have the lesion, if so, jump into a volcano, otherwise don’t”. Should you infer the causal graph
lesion -> (EDT decision: jump?) -> diewith a perfect correlation betweenlesionandjump? (Hint: no, that would be stupid, since we’re not using jump-based-on-lesion-decision-theory, we’re using EDT.)In programming, we also say “garbage in, garbage out”. You are feeding EDT garbage input by giving it factually wrong joint probability distributions.
Ok, what about cases where there are multiple causal hypotheses that are observationally indistinguishable:
a → b → c
vs
a ← b ← c
Both models imply the same joint probability distribution p(a,b,c) with a single conditional independence (a independent of c given b) and cannot be told apart without experimentation. That is, you cannot call p(a,b,c) “factually wrong” because the correct causal model implies it. But the wrong causal model implies it too! To figure out which is which requires causal information. You can give it to EDT and it will work—but then it’s not EDT anymore.
I can give you a graph which implies the same independences as my HAART example but has a completely different causal structure, and the procedure you propose here:
http://lesswrong.com/lw/hwq/evidential_decision_theory_selection_bias_and/9d6f
will give the right answer in one case and the wrong answer in another.
The point is, EDT lacks a rich enough input language to avoid getting garbage inputs in lots of standard cases. Or, more precisely, EDT lacks a rich enough input languages to tell when input is garbage and when it isn’t. This is why EDT is a terrible decision theory.
I think there are a couple of confusions this sentence highlights.
First, there are approaches to solving decision theory problems that don’t use causal models. Part of what has made this conversation challenging is that there are several different ways to represent the world- and so even if CDT is the best / natural one, it needs to be distinguished from other approaches. EDT is not CDT in disguise; the two are distinct formulas / approaches.
Second, there are good reasons to modularize the components of the decision theory, so that you can treat learning a model from data separately from making a decision given a model. An algorithm to turn models into decisions should be able to operate on an arbitrary model, where it sees a → b → c as isomorphic to Drunk → Fall → Death.
To tell an anecdote, when my decision analysis professor would teach that subject to petroleum engineers, he quickly learned not to use petroleum examples. Say something like “suppose the probability of striking oil by drilling a well here is 40%” and an engineer’s hand will shoot up, asking “what kind of rock is it?”. The kind of rock is useful for determining whether or not the probability is 40% or something else, but the question totally misses the point of what the professor is trying to teach. The primary example he uses is choosing a location for a party subject to the uncertainty of the weather.
I’m not sure how to interpret this sentence.
The way EDT operates is to perform the following three steps for each possible action in turn:
Assume that I saw myself doing X.
Perform a Bayesian update on this new evidence.
Calculate and record my utility.
It then chooses the possible action which had the highest calculated utility.
One interpretation is you saying that EDT doesn’t make sense, but I’m not sure I agree with what seems to be the stated reason. It looks to me like you’re saying “it doesn’t make sense to assume that you do X until you know what you decide!”, when I think that does make sense, but the problem is using that assumption as Bayesian evidence as if it were an observation.
Ideal Bayesian updates assume logical omniscience, right? Including knowledge about logical fact of what EDT would do for any given input. If you know that you are an EDT agent, and condition on all of your past observations and also on the fact that you do X, but X is not in fact what EDT does given those inputs, then as an ideal Bayesian you will know that you’re conditioning on something impossible. More generally, what update you perform in step 2 depends on EDT’s input-output map, thus making the definition circular.
So, is EDT really underspecified? Or are you supposed to search for a fixed point of the circular definition, if there is one? Or does it use some method other than Bayes for the hypothetical update? Or does an EDT agent really break if it ever finds out its own decision algorithm? Or did I totally misunderstand?
Note that step 1 is “Assume that I saw myself doing X,” not “Assume that EDT outputs X as the optimal action.” I believe that excludes any contradictions along those lines. Does logical omniscience preclude imagining counterfactual worlds?
If I already know “I am EDT”, then “I saw myself doing X” does imply “EDT outputs X as the optimal action”. Logical omniscience doesn’t preclude imagining counterfactual worlds, but imagining counterfactual worlds is a different operation than performing Bayesian updates. CDT constructs counterfactuals by severing some of the edges in its causal graph and then assuming certain values for the nodes that no longer have any causes. TDT does too, except with a different graph and a different choice of edges to sever.
I don’t know how I can fail to communicate so consistently.
Yes, you can technically apply “EDT” to any causal model or (more generally) joint probability distribution containing a “EDT agent decision” node. But in practice this freedom is useless, because to derive an accurate model you generally need to take account of a) the fact that the agent is using EDT and b) any observations the agent does or does not make. To be clear, the input EDT requires is a probabilistic model describing the EDT agent’s situation (not describing historical data of “similar” situations).
There are people here trying to argue against EDT by taking a model describing historical data (such as people following dumb decision theories jumping into volcanoes) and feeding this model directly into EDT. Which is simply wrong. A model that describes the historical behaviour of agents using some other decision theory does not in general accurately describe an EDT agent in the same situation.
The fact that this egregious mistake looks perfectly normal is an artifact of the fact that CDT doesn’t care about causal parents of the “CDT decision” node.
I suspect it’s because what you are referring to as “EDT” is not what experts in the field use that technical term to mean.
nsheppard-EDT is, as far as I can tell, the second half of CDT. Take a causal model and use the do() operator to create the manipulated subgraph that would result taking possible action (as an intervention). Determine the joint probability distribution from the manipulated subgraph. Condition on observing that action with the joint probability distribution, and calculate the probabilistically-weighted mean utility of the possible outcomes. This is isomorphic to CDT, and so referring to it as EDT leads to confusion.
Whatever. I give up.
Here’s a modified version. Instead of a smoking lesion, there’s a “jump into active volcano lesion”. Furthermore, the correlation isn’t as puny as for the smoking lesion. 100% of people with this lesion jump into active volcanoes and die, and nobody else does.
Should you go jump into an active volcano?
Using a decision theory to figure out what decision you should make assumes that you’re capable of making a decision. “The lesion causes you to jump into an active volcano/smoke” and “you can choose whether to jump into an active volcano/smoke” are contradictory. Even “the lesion is correlated (at less than 100%) with jumping into an active volcano/smoking” and “you can choose whether to jump into an active volcano/smoke” are contradictory unless “is correlated with” involves some correlation for people who don’t use decision theory and no correlation for people who do.
Agreed.
Doesn’t this seem sort of realistic, actually? Decisions made with System 1 and System 2, to use Kahneman’s language, might have entirely different underlying algorithms. (There is some philosophical trouble about how far we can push the idea of an ‘intervention’, but I think for human-scale decisions there is a meaningful difference between interventions and observations such that CDT distinguishing between them is a feature.)
This maps onto an objection by proponents of EDT that the observational data might not be from people using EDT, and thus the correlation may disappear when EDT comes onto the stage. I think that objection proves too much- suppose all of our observational data on the health effects of jumping off cliffs comes from subjects who were not using EDT (suppose they were drunk). I don’t see a reason inside the decision theory for differentiating between the effects of EDT on the correlation between jumping off the cliff and the effects of EDT on the correlation between smoking and having the lesion.
These two situations correspond to two different causal structures—Drunk → Fall → Death and Smoke ← Lesion → Cancer—which could have the same joint probability distribution. The directionality of the arrow is something that CDT can make use of to tell that the two situations will respond differently to interventions at Drunk and Smoke: it is dangerous to be drunk around cliffs, but not to smoke (in this hypothetical world).
EDT cannot make use of those arrows. It just has Drunk—Fall—Death and Smoke—Lesion—Cancer (where it knows that the correlations between Drunk and Death are mediated by Fall, and the correlations between Smoke and Cancer are mediated by Lesion). If we suppose that adding an EDT node might mean that the correlation between Smoke and Lesion (and thus Cancer) might be mediated by EDT, then we must also suppose that adding an EDT node might mean that the correlation between Drunk and Fall (and thus Death) might be mediated by EDT.
(I should point out that the EDT node describes whether or not EDT was used to decide to drink, not to decide whether or not to fall off the cliff, by analogy of using EDT to decide whether or not to smoke, rather than deciding whether or not to have a lesion.)