Might you be willing to explain for the rest of us what the “faithfulness assumption” is, and why it’s needed for “no correlation” to imply “no causation”? I’d appreciate it.
Absolutely! In a typical Bayesian network, we represent a set of probability distributions by a directed acyclic graph, such that any distribution in the set can be written as
p[x1,…,xn]=∏ip[xi|pa[xi]]
in other words, for every random variable in the distribution we associate a node in the graph, and the distribution can be factorized into a product of conditional densities, where each conditional is a variable (node) conditional on variables corresponding to parents of the node.
This implies that if certain types of paths in the graph from a node set X to a node set Y are “blocked” in a particular way (e.g. d-separated) by a third set Z, then in all densities that factorize as above, X is independent of Y conditional on Z. Note that this implication is one way. In particular, we can still have some conditional independence that just happens to hold because the numbers in the distribution lined up just right, and the graph does not in fact advertise this independence via d-separation. When this happens, we say the distribution is unfaithful to the graph.
If we pick parameters of a distribution that factorizes at random then almost all parameter picks will be faithful to the graph. However, lots of distributions are “near unfaithful”, that is they are faithful, but it’s hard to tell with limited samples. Moreover, we can’t tell in advance how many samples we need to tell. Also, it’s easy to construct faithfulness violations and they do occur in practice. For example, we may have an AIDS drug that suppresses the HIV (so it really does help!), but the drug is very nasty, with lots of side effects and so on, so doctors usually wait until the patient is already very sick before giving the drug. If we then look at associations between instances of the use of this drug we may well find that those who take the drug either die more (positive association of drug with death!) or don’t die less frequently than those without the drug (no association of drug with death!).
Does this then mean the drug has a bad effect or no effect? No! It just means there is an obvious confounder of health status that we aren’t recording. In the second case, this confounder is causing the distribution over drug and death to be “unfaithful”: there is an arrow from drug to death, but there is no dependence of death on drug. And yet there is still a causal effect.
Note: I am glossing over some distinctions between a Bayesian network and a causal model in order to not muddy the discussion. What is important to note is that: (a) A Bayesian network is not a graphical causal model, but (b) a graphical causal model induces a Bayesian network on the observable data. Faithfulness (or lack of it) applies to the network appearing due to (b), and thus affects causal reasoning in the underlying causal model.
This seems like a big problem for inferring “no causation” from “no correlation.” Is there a standard methodological solution? And, do researchers often just choose to infer “no causation” from “no correlation” and hope for the best, or do they avoid inferring “no causation” from “no correlation” due to the fact that they can’t tell whether the faithfulness assumption holds?
Well, in some sense this is why causal inference is hard. Most of the time if you see independence that really does mean there is nothing there. The reasonable default is the null hypothesis: there is no causal effect. However, if you are poking around because you suspect there is something there, then not seeing any correlations does not mean you should give up. What it does mean is you should think about causal structure and specifically about confounders.
What people do about confounders is:
(a) Try to measure them somehow (epidemiology, medicine). If you can measure confounders you can adjust for them, and then the effect cancellation will go away.
(b) Try to find an instrumental variable (econometrics). If you can find a good instrument, you can get a causal effect with some parametric assumptions, even if there are unmeasured confounders.
(c) Try to randomize (statistics). This explicitly cuts out all confounding.
(d) You can sometimes get around unmeasured confounders by using strong mediating variables by means of “front-door” type methods. These methods aren’t really well known, and aren’t commonly used.
There is no royal road: getting rid of confounders is the entire point of causal inference. People have been thinking of clever ways to do it for close to a hundred years now. If you have infinite samples, and know where unobserved confounding is, there is an algorithm for getting the causal effect from observational data by being sneaky. This algorithm only succeeds sometimes, and if it doesn’t, there is no other way in general to do it (e.g. it’s “complete”). More in my thesis, if you are curious.
One more question, since this is your field. Do you happen to know of an instance where some new causal effect was discovered from observational data via causal modeling, and this cause was later confirmed by an RCT?
Well, I think smoking/cancer was first established in case control studies. In general people move up the “hierarchy of evidence” Kawoomba mentioned. At the end of the day, people only trust RCTs (and they are right, other methods rely on more assumptions). There is another good example, but let me double check before posting.
With case control studies you have the additional problem of selection bias, on top of confounding.
Right, you can’t always RCT in humans. But a causal mechanism + RCTs in animals biologically close to humans is convincing for something like lung cancer where minor differences among mammals shouldn’t matter much (although e.g. bears have evolved some crazy stuff to deal with all that fat they eat before hibernating).
where minor differences among mammals shouldn’t matter much
I think you are entirely optimistic. I recently pointed out that the research indicates that animal studies routinely (probably usually) do not transfer, and as it happens, animal smoking studies are an example of this, according to Hanson. So the differences are often far from minor, and even if there were cancer in the animal studies, we could infer very little from it.
I find much to agree with in Hanson’s writings, but in this case I just don’t find him convincing. One issue is that cancer is a scourge of a long-living animal. One hypothesis is that smoking causes long term cumulative damage, and you might not see effects in mice or dogs because they die too soon regardless. There is also the issue that we have a fair idea of the carcinogenic mechanism now, so if you think smoking does not cause harm, there also needs to be a story how that mechanism is foiled in humans.
I find much to agree with in Hanson’s writings, but in this case I just don’t find him convincing.
His interpretation, or his evidence? I point this out because it looks to me like your position has shifted from “the smoking / lung cancer link is established by RCTs in animals” to “even though RCTs don’t establish the smoking / lung cancer link for animals, we have other reasons to believe in the smoking / lung cancer link for humans.”
I find much to agree with in Hanson’s writings, but in this case I just don’t find him convincing...One hypothesis is that smoking causes long term cumulative damage, and you might not see effects in mice or dogs because they die too soon regardless.
So: heads I win, tails you lose? If the studies had found smoking caused cancer in animals, well, that proves it! And if they don’t, well, that just means they didn’t run long enough so we can ignore them and say we “just don’t find them convincing”...
There is also the issue that we have a fair idea of the carcinogenic mechanism now, so if you think smoking does not cause harm, there also needs to be a story how that mechanism is foiled in humans.
You don’t think there were plenty of ‘fair ideas’ of mechanisms floating around in the thousands of animal studies and interventions covered in my animal studies link? Any researcher worth his degree can come up with a plausible ex post explanation.
This algorithm only succeeds sometimes, and if it doesn’t, there is no other way in general to do it (e.g. it’s “complete”). More in my thesis, if you are curious.
Your thesis deals only with acyclic causal graphs. What is the current state of the art for cyclic causal graphs? You’ll know already that I’ve been looking at that, and I have various papers of other people that attempt to take steps in that direction, but my impression is that none of them actually get very far and there is nothing like a set of substantial results that one can point to. Even my own, were they in print yet, are primarily negative.
(a) Can’t assign Pearlian semantics to cyclic graphs.
(b) If you assign equilibrium semantics, you might as well use a dynamic causal Bayesian network, a cyclic graph does not buy you anything.
(c) A graph representing the Markov property of the equilibrium distribution of a Markov chain represented by a causal DBN is an interesting open question. (This graph wouldn’t have a causal interpretation of course).
As far as I can tell, epidemiology and medicine are mostly doing (c), in the form of RCTs (which are the gold standard of medical evidence, other than meta-reviews). There are other study designs such as most variants of case-control studies and cohort studies which do take the (a) approach, but they aren’t considered to be the same level of evidence as randomized controlled trials.
but they aren’t considered to be the same level of evidence as randomized controlled trials.
Quite rightly—if we randomize, we don’t care what the underlying causal structure is, we just cut all confounding out anyways. Methods (a), (b), (d) all rely on various structural assumptions that may or may not hold. However, even given those assumptions figuring out how to do causal inference from observational data is quite difficult. The problem with RCTs is expense, ethics, and statistical power (hard to enroll a ton of people in an RCT).
Epidemiology and medicine does a lot of (a), look for the keywords “g-formula”, “g-estimation”, “inverse probability weighting,” “propensity score”, “marginal structural models,” “structural nested models”, “covariate adjustment,” “back-door criterion”, etc. etc.
People talk about “controlling for other factors” when discussing associations all the time, even in non-technical press coverage. They are talking about (a).
People talk about “controlling for other factors” when discussing associations all the time, even in non-technical press coverage. They are talking about (a).
True, true. “Gold standard” or “preferred level of evidence” versus “what’s mostly conducted given the funding limitations”. However, to make it into a guideline, there are often RCT follow-ups for hopeful associations uncovered by the lesser study designs.
look for the keywords “g-formula”, “g-estimation”, “inverse probability weighting,” “propensity score”, “marginal structural models,” “structural nested models”, “covariate adjustment,” “back-door criterion”, etc. etc.
I, of course, know all of those. The letters, I mean.
“No subtle confounders” and “increasing sample size (decreases relevance and likelihood of such special cases)” would have m-answered your previous z-comments. (SCNR)
Might you be willing to explain for the rest of us what the “faithfulness assumption” is, and why it’s needed for “no correlation” to imply “no causation”? I’d appreciate it.
Absolutely! In a typical Bayesian network, we represent a set of probability distributions by a directed acyclic graph, such that any distribution in the set can be written as
p[x1,…,xn]=∏ip[xi|pa[xi]]
in other words, for every random variable in the distribution we associate a node in the graph, and the distribution can be factorized into a product of conditional densities, where each conditional is a variable (node) conditional on variables corresponding to parents of the node.
This implies that if certain types of paths in the graph from a node set X to a node set Y are “blocked” in a particular way (e.g. d-separated) by a third set Z, then in all densities that factorize as above, X is independent of Y conditional on Z. Note that this implication is one way. In particular, we can still have some conditional independence that just happens to hold because the numbers in the distribution lined up just right, and the graph does not in fact advertise this independence via d-separation. When this happens, we say the distribution is unfaithful to the graph.
If we pick parameters of a distribution that factorizes at random then almost all parameter picks will be faithful to the graph. However, lots of distributions are “near unfaithful”, that is they are faithful, but it’s hard to tell with limited samples. Moreover, we can’t tell in advance how many samples we need to tell. Also, it’s easy to construct faithfulness violations and they do occur in practice. For example, we may have an AIDS drug that suppresses the HIV (so it really does help!), but the drug is very nasty, with lots of side effects and so on, so doctors usually wait until the patient is already very sick before giving the drug. If we then look at associations between instances of the use of this drug we may well find that those who take the drug either die more (positive association of drug with death!) or don’t die less frequently than those without the drug (no association of drug with death!).
Does this then mean the drug has a bad effect or no effect? No! It just means there is an obvious confounder of health status that we aren’t recording. In the second case, this confounder is causing the distribution over drug and death to be “unfaithful”: there is an arrow from drug to death, but there is no dependence of death on drug. And yet there is still a causal effect.
Note: I am glossing over some distinctions between a Bayesian network and a causal model in order to not muddy the discussion. What is important to note is that: (a) A Bayesian network is not a graphical causal model, but (b) a graphical causal model induces a Bayesian network on the observable data. Faithfulness (or lack of it) applies to the network appearing due to (b), and thus affects causal reasoning in the underlying causal model.
Thanks!
This seems like a big problem for inferring “no causation” from “no correlation.” Is there a standard methodological solution? And, do researchers often just choose to infer “no causation” from “no correlation” and hope for the best, or do they avoid inferring “no causation” from “no correlation” due to the fact that they can’t tell whether the faithfulness assumption holds?
Well, in some sense this is why causal inference is hard. Most of the time if you see independence that really does mean there is nothing there. The reasonable default is the null hypothesis: there is no causal effect. However, if you are poking around because you suspect there is something there, then not seeing any correlations does not mean you should give up. What it does mean is you should think about causal structure and specifically about confounders.
What people do about confounders is:
(a) Try to measure them somehow (epidemiology, medicine). If you can measure confounders you can adjust for them, and then the effect cancellation will go away.
(b) Try to find an instrumental variable (econometrics). If you can find a good instrument, you can get a causal effect with some parametric assumptions, even if there are unmeasured confounders.
(c) Try to randomize (statistics). This explicitly cuts out all confounding.
(d) You can sometimes get around unmeasured confounders by using strong mediating variables by means of “front-door” type methods. These methods aren’t really well known, and aren’t commonly used.
There is no royal road: getting rid of confounders is the entire point of causal inference. People have been thinking of clever ways to do it for close to a hundred years now. If you have infinite samples, and know where unobserved confounding is, there is an algorithm for getting the causal effect from observational data by being sneaky. This algorithm only succeeds sometimes, and if it doesn’t, there is no other way in general to do it (e.g. it’s “complete”). More in my thesis, if you are curious.
Thanks again.
One more question, since this is your field. Do you happen to know of an instance where some new causal effect was discovered from observational data via causal modeling, and this cause was later confirmed by an RCT?
Well, I think smoking/cancer was first established in case control studies. In general people move up the “hierarchy of evidence” Kawoomba mentioned. At the end of the day, people only trust RCTs (and they are right, other methods rely on more assumptions). There is another good example, but let me double check before posting.
With case control studies you have the additional problem of selection bias, on top of confounding.
I thought there were still no actual RCTs of smoking in humans.
Right, you can’t always RCT in humans. But a causal mechanism + RCTs in animals biologically close to humans is convincing for something like lung cancer where minor differences among mammals shouldn’t matter much (although e.g. bears have evolved some crazy stuff to deal with all that fat they eat before hibernating).
I think you are entirely optimistic. I recently pointed out that the research indicates that animal studies routinely (probably usually) do not transfer, and as it happens, animal smoking studies are an example of this, according to Hanson. So the differences are often far from minor, and even if there were cancer in the animal studies, we could infer very little from it.
Out of curiosity, do you smoke?
No.
I find much to agree with in Hanson’s writings, but in this case I just don’t find him convincing. One issue is that cancer is a scourge of a long-living animal. One hypothesis is that smoking causes long term cumulative damage, and you might not see effects in mice or dogs because they die too soon regardless. There is also the issue that we have a fair idea of the carcinogenic mechanism now, so if you think smoking does not cause harm, there also needs to be a story how that mechanism is foiled in humans.
His interpretation, or his evidence? I point this out because it looks to me like your position has shifted from “the smoking / lung cancer link is established by RCTs in animals” to “even though RCTs don’t establish the smoking / lung cancer link for animals, we have other reasons to believe in the smoking / lung cancer link for humans.”
So: heads I win, tails you lose? If the studies had found smoking caused cancer in animals, well, that proves it! And if they don’t, well, that just means they didn’t run long enough so we can ignore them and say we “just don’t find them convincing”...
You don’t think there were plenty of ‘fair ideas’ of mechanisms floating around in the thousands of animal studies and interventions covered in my animal studies link? Any researcher worth his degree can come up with a plausible ex post explanation.
Your thesis deals only with acyclic causal graphs. What is the current state of the art for cyclic causal graphs? You’ll know already that I’ve been looking at that, and I have various papers of other people that attempt to take steps in that direction, but my impression is that none of them actually get very far and there is nothing like a set of substantial results that one can point to. Even my own, were they in print yet, are primarily negative.
The recent stuff I have seen is negative results:
(a) Can’t assign Pearlian semantics to cyclic graphs.
(b) If you assign equilibrium semantics, you might as well use a dynamic causal Bayesian network, a cyclic graph does not buy you anything.
(c) A graph representing the Markov property of the equilibrium distribution of a Markov chain represented by a causal DBN is an interesting open question. (This graph wouldn’t have a causal interpretation of course).
As far as I can tell, epidemiology and medicine are mostly doing (c), in the form of RCTs (which are the gold standard of medical evidence, other than meta-reviews). There are other study designs such as most variants of case-control studies and cohort studies which do take the (a) approach, but they aren’t considered to be the same level of evidence as randomized controlled trials.
Quite rightly—if we randomize, we don’t care what the underlying causal structure is, we just cut all confounding out anyways. Methods (a), (b), (d) all rely on various structural assumptions that may or may not hold. However, even given those assumptions figuring out how to do causal inference from observational data is quite difficult. The problem with RCTs is expense, ethics, and statistical power (hard to enroll a ton of people in an RCT).
Epidemiology and medicine does a lot of (a), look for the keywords “g-formula”, “g-estimation”, “inverse probability weighting,” “propensity score”, “marginal structural models,” “structural nested models”, “covariate adjustment,” “back-door criterion”, etc. etc.
People talk about “controlling for other factors” when discussing associations all the time, even in non-technical press coverage. They are talking about (a).
True, true. “Gold standard” or “preferred level of evidence” versus “what’s mostly conducted given the funding limitations”. However, to make it into a guideline, there are often RCT follow-ups for hopeful associations uncovered by the lesser study designs.
I, of course, know all of those. The letters, I mean.
“No subtle confounders” and “increasing sample size (decreases relevance and likelihood of such special cases)” would have m-answered your previous z-comments. (SCNR)