Good question! “Untestable assumption” can actually mean two different things:
In this context, you are correct to point out that I am talking about assumptions that are not testable by the data we are analyzing. I would be able to falsify my unconfoundedness assumption if I run an experiment where I first observe what value the treatment variable would take naturally in all individuals, then intervene so that everyone are treated, and look at whether the distribution of the outcome differs between the group who naturally would have had treatment, and the group who naturally would not have had treatment
In other contexts, there are other types of untestable assumptions, which are unfalsifiable even in principle. These relate to independences between counterfactual variables from different “worlds”. Basically, they assume that certain columns in your ideal dataset are independent of each other, when it is impossible even in theory to observe those two columns in the whole population at the same time.
If you refuse to make assumptions of the second type, you will still be able to estimate the effect of any intervention that is identifiable in Pearl’s causal framework NPSEM, but you will not be able to analyze mediation or causal pathways. This is the difference between Pearl’s model NPSEM and Robins’ model FFRCISTG. The refusal to make unfalsifiable assumptions about independences between cross-world counterfactuals is also the primary motivation behind the “Single World Intervention Graph” paper by Robins and Richardson, which Ilya linked to in another comment in this thread.
Good post, thanks. FFRCISTG still assumes SUTVA, which is untestable (also, like any structural equation model, it assumes absent arrows represent absence of individual level effects, which seems like it is also untestable (?)).
. . . someone should write up an explanation and post it here. :)
Good question! “Untestable assumption” can actually mean two different things:
In this context, you are correct to point out that I am talking about assumptions that are not testable by the data we are analyzing. I would be able to falsify my unconfoundedness assumption if I run an experiment where I first observe what value the treatment variable would take naturally in all individuals, then intervene so that everyone are treated, and look at whether the distribution of the outcome differs between the group who naturally would have had treatment, and the group who naturally would not have had treatment
In other contexts, there are other types of untestable assumptions, which are unfalsifiable even in principle. These relate to independences between counterfactual variables from different “worlds”. Basically, they assume that certain columns in your ideal dataset are independent of each other, when it is impossible even in theory to observe those two columns in the whole population at the same time.
If you refuse to make assumptions of the second type, you will still be able to estimate the effect of any intervention that is identifiable in Pearl’s causal framework NPSEM, but you will not be able to analyze mediation or causal pathways. This is the difference between Pearl’s model NPSEM and Robins’ model FFRCISTG. The refusal to make unfalsifiable assumptions about independences between cross-world counterfactuals is also the primary motivation behind the “Single World Intervention Graph” paper by Robins and Richardson, which Ilya linked to in another comment in this thread.
Good post, thanks. FFRCISTG still assumes SUTVA, which is untestable (also, like any structural equation model, it assumes absent arrows represent absence of individual level effects, which seems like it is also untestable (?)).
. . . someone should write up an explanation and post it here. :)