Generalizing Experimental Results by Leveraging Knowledge of Mechanisms
In a recent post (and papers), Anders Huitfeldt and co-authors have discussed ways of achieving external validity in the presence of “effect heterogeneity.” These results are not immediately inferable using a standard (non-parametric) selection diagram, which has led them to conclude that selection diagrams may not be helpful for “thinking more closely about effect heterogeneity” and, thus, might be “throwing the baby out with the bathwater.”
Taking a closer look at the analysis of Anders and co-authors, and using their very same examples, we came to quite different conclusions. In those cases, transportability is not immediately inferable in a fully nonparametric structural model for a simple reason: it relies on functional constraints on the structural equation of the outcome. Once these constraints are properly incorporated in the analysis, all results flow naturally from the structural model, and selection diagrams prove to be indispensable for thinking about heterogeneity, for extrapolating results across populations, and for protecting analysts from unwarranted generalizations. See details in the note we post here for discussion.
That’s a really nice piece. It shows how to formulate the relevant background knowledge in the graphical language, how to derive the intuitive results which Huitfeldt et al predicted, and how the structural formulation correctly predicts when the intuitive results fail. Well done.
Thanks, John.
Welcome to LessWrong Carlos and Judea :) I’m excited about your response to Anders’ paper and to Anders + John’s discussion.
As a site as a whole, lots of users (as well as Anders and John) are really interested in the sorts of work you do, relating to causal modelling. For example, one user wrote an introduction to causal modelling for a general audience, and other users have reviewed books like Causality and The Book of Why.
I also want to let you know that you can use LaTex in your comments and posts if you want to, using cmd-4 / ctrl-4.
You can write things like S(ω)=αg2ω5e[−0.74{ωUω19.5g}−4] .
Let me know if you need any technical/editor support. You can reply here, or also use the Intercom in the bottom-right corner of the screen :)
Thank you, Ben. Nice to see that people are interested in causal modeling here as well!
As an update, “Generalizing Experimental Results by Leveraging Knowledge of Mechanisms” has been accepted for publication. Which is good, because the link that @carlos_cinelli posted is now broken.