Yes, thank you. Strong upvote. In a previous version I’d had I noticed this yesterday before going to sleep, let me fix the code and run it again (and update the post).
In a previous version I did use actual causal influence, not just direct nodes. The fixed code then is
Interestingly, this makes the number of causal relationships higher, and should therefore also increase the number of causal non-correlations (which is also what now running the code suggests).
I think the transitive closure captures “A is causally upstream of B” and “B is causally upstream of A” but not “Some common thing C is causally upstream of both A and B”. Going back to the example in the above post:
if I do
sem = nx.DiGraph()
coefficients = {(0, 1): -0.371, (1, 5): +0.685, (2, 3): +1.139, (2, 4): -0.332, (4, 5): +0.580}
sem.add_nodes_from(range(6))
for (src, dst), weight in coefficients.items():
sem.add_edge(src, dst, label=f'{weight:+.2f}')
print([(i, j) for i, j in nx.transitive_closure(sem).edges()])
However, we would expect nonzero correlation between the values of e.g. node 3 and node 4, because both 3 and 4 are causally downstream of 2. However, the transitive closure is missing (3, 4) and (3, 5).
There might be a cleaner and more “mathy” way of saying “all pairs of nodes a and b such that the intersection of (a and all a‘s ancestors) and (b and all b’s ancestors) is non-empty”, but if there is I don’t know the math term for it. Still, I think that is the construct you need here.
If some common variable C is causally upstream both of A and B, then I wouldn’t say that A causes B, or B causes A—intervening on A can’t possibly change B, and intervening on B can’t change A (which is the understanding of causation by Pearl).
If some common variable C is causally upstream both of A and B, then I wouldn’t say that A causes B, or B causes A—intervening on A can’t possibly change B, and intervening on B can’t change A (which is the understanding of causation by Pearl).
I agree with this. And yet.
I, however, have an inner computer scientist. And he demands answers. He will not rest until he knows how often ¬Correlation ⇒ ¬Causation, and how often it doesn’t. [...] Let’s take all the correlations between variables which don’t have any causal relationship. The largest of those is the “largest uncaused correlation”. Correlations between two variables which cause each other but are smaller than the largest uncaused correlation are “too small”: There is a causation but it’s not detected.
The issue with that is that your “largest uncaused correlation” can be arbitrarily large—if you’ve got some common factor C that explains 99% of the variance in downstream things A and B, but A does not affect B and B does not affect A, your largest uncaused correlation is going to be > 0.9 and as such you’ll think that any correlations less than 0.9 are fake / undetected.
Let’s make the above diagram concrete:
Let’s consider the following causal influence graph
0: Past 24-hour rainfall at SEA-TAC airport
1: Electricity spot price in Seattle (Seattle gets 80% of its electricity from hydro power)
2: Average electric car cost in US, USD
3: Total value of vehicle registration fees collected in California (California charges an amount proportional to the value of the car)
4: Fraction of households with an electric car in Seattle
5: Average household electric bill in Seattle
Changing the total value of vehicle registration fees collected in California will not affect the fraction of households with an electric car in Seattle, nor will changing the fraction of households with an electric car in Seattle affect the total value of vehicle registration fees collected in California. And yet we expect a robust correlation between those two.
Whether or not we can tell that past 24 hour rainfall causes changes in the spot price of electricity should not depend on the relationship between vehicle registration fees in California and electric vehicle ownership in Seattle.
Yes, thank you. Strong upvote. In a previous version I’d had I noticed this yesterday before going to sleep, let me fix the code and run it again (and update the post).
In a previous version I did use actual causal influence, not just direct nodes. The fixed code then is
instead of
Interestingly, this makes the number of causal relationships higher, and should therefore also increase the number of causal non-correlations (which is also what now running the code suggests).
I think the transitive closure captures “A is causally upstream of B” and “B is causally upstream of A” but not “Some common thing C is causally upstream of both A and B”. Going back to the example in the above post:
if I do
then I get
However, we would expect nonzero correlation between the values of e.g. node
3and node4, because both3and4are causally downstream of2. However, the transitive closure is missing(3, 4)and(3, 5).There might be a cleaner and more “mathy” way of saying “all pairs of nodes
aandbsuch that the intersection of (aand alla‘s ancestors) and (band allb’s ancestors) is non-empty”, but if there is I don’t know the math term for it. Still, I think that is the construct you need here.If some common variable C is causally upstream both of A and B, then I wouldn’t say that A causes B, or B causes A—intervening on A can’t possibly change B, and intervening on B can’t change A (which is the understanding of causation by Pearl).
I agree with this. And yet.
The issue with that is that your “largest uncaused correlation” can be arbitrarily large—if you’ve got some common factor C that explains 99% of the variance in downstream things A and B, but A does not affect B and B does not affect A, your largest uncaused correlation is going to be > 0.9 and as such you’ll think that any correlations less than 0.9 are fake / undetected.
Let’s make the above diagram concrete:
Let’s consider the following causal influence graph
0: Past 24-hour rainfall at SEA-TAC airport
1: Electricity spot price in Seattle (Seattle gets 80% of its electricity from hydro power)
2: Average electric car cost in US, USD
3: Total value of vehicle registration fees collected in California (California charges an amount proportional to the value of the car)
4: Fraction of households with an electric car in Seattle
5: Average household electric bill in Seattle
Changing the total value of vehicle registration fees collected in California will not affect the fraction of households with an electric car in Seattle, nor will changing the fraction of households with an electric car in Seattle affect the total value of vehicle registration fees collected in California. And yet we expect a robust correlation between those two.
Whether or not we can tell that past 24 hour rainfall causes changes in the spot price of electricity should not depend on the relationship between vehicle registration fees in California and electric vehicle ownership in Seattle.