When is correlation transitive?

It’s a well-known property of correlation that it’s not transitive in general. If $X, Y, Z$ are three real-valued random variables such that $ρ (X, Y) > 0$ and $ρ (Y, Z) > 0$ , it doesn’t have to be the case that $ρ (X, Z) > 0$ .

Nevertheless, there are some circumstances under which correlation is transitive. I will focus on two such cases in this post.

Primer: correlation as an inner product

For what follows, some background knowledge is necessary that we can regard correlations of real-valued random variables with finite second moments as inner products in an appropriate Hilbert space.

Specifically, if $X, Y$ are two such random variables with zero mean and unit standard deviation, which is a simplification we can always make as correlation is invariant under translation and scalar multiplication, then we can compute

$ρ (X, Y) = \frac{cov (X, Y)}{σ_{X} σ_{Y}} = E [X Y] - E [X] E [Y] = E [X Y]$

The pairing $(A, B) \to E [A B]$ defines an inner product on the space of random variables with finite second moments where two random variables are considered equivalent if they are equal with probability $1$ (almost surely). The properties that we expect out of an inner product are easy to check: the pairing is obviously bilinear and positive definite.

Furthermore, it turns out this inner product turns the space of random variables with finite second moments into a Hilbert space: the vector space turns out to be complete under the induced norm $∥ X ∥ = \sqrt{E [X^{2}]}$ . Roughly speaking, this means that we can take orthogonal projections onto closed subspaces with impunity.

Now that we have this framework, we can move on to the main results of this post.

Correlation is transitive when the correlations are sufficiently strong

I’ll first prove the following:

Claim 1: If $ρ (X, Y) = a$ and $ρ (Y, Z) = b$ , then

$a b - \sqrt{(1 - a^{2}) (1 - b^{2})} \leq ρ (X, Z) \leq a b + \sqrt{(1 - a^{2}) (1 - b^{2})}$

Moreover, these bounds are tight: for any $a, b$ , there is a combination $X, Y, Z$ for which we can make either the right or the left inequality into an equality.

Proof

We can assume $X, Y, Z$ have mean zero and unit variance without loss of generality. Taking orthogonal projections of $X, Z$ onto the one-dimensional subspace spanned by $Y$ , we can write

$X = a Y + \sqrt{1 - a^{2}} H_{X Y} Z = b Y + \sqrt{1 - b^{2}} H_{Z Y}$

where $E [Y H_{X Y}] = E [Y H_{Z Y}] = 0$ and the random variables $H_{X Y}, H_{Z Y}$ have mean zero and variance $1$ . Taking inner products gives

$E [X Z] = a b + \sqrt{(1 - a^{2}) (1 - b^{2})} E [H_{X Y} H_{Z Y}]$

Using the Cauchy-Schwarz inequality for our inner product finishes the proof: $| E [H_{X Y} H_{Z Y}] | \leq ∥ H_{X Y} ∥ \cdot ∥ H_{Z Y} ∥ = 1$ . For the existence proof, let $Y$ be an arbitrary random variable with mean zero and unit variance and pick $H_{X Y}, H_{Y Z}$ to be perfectly correlated or perfectly anti-correlated standard Gaussians that are uncorrelated with $Y$ .

Interpretation

When $a, b$ are large and positive, the lower bound $a b - \sqrt{(1 - a^{2}) (1 - b^{2})}$ is also positive, and so we have a guaranteed positive correlation between $X$ and $Z$ .

One way to simplify this is to make it single-dimensional by assuming $a = b$ . In this case, the lower bound is $2 a^{2} - 1$ . If we want a guaranteed positive correlation between $X$ and $Z$ , this means the correlations $ρ (X, Y) = ρ (Y, Z) = a$ have to satisfy $a > 1 / \sqrt{2} \approx 0.7$ .

This condition is quite strict, and we might wonder if some transitivity of correlation can be recovered in the absence of such strong correlations between $X, Y$ and $Y, Z$ . It turns out the answer is yes, at least if we assume the random variables are in some sense “generic”.

Correlation is transitive on the average

It turns out that in a suitable sense, when $X, Y$ and $Y, Z$ are positively correlated, there is a tendency for $X, Z$ to also be positively correlated, even though per Claim 1 we can’t deduce that they must be positively correlated.

The precise version of this claim is as follows:

Claim 2: Let $X, Y, Z$ be vectors independently and uniformly distributed on the $n$ -dimensional unit sphere $S^{n} \subset R^{n + 1}$ , and let $- 1 \leq a, b \leq 1$ be two real numbers. Denote the standard inner product on $R^{n + 1}$ by $(x, y) \to ⟨ x, y ⟩$ . Then, we have

$E [⟨ X, Z ⟩ | ⟨ X, Y ⟩ = a, ⟨ Y, Z ⟩ = b] = a b$

In intuitive terms, this claim is saying that if $X, Y, Z$ are “generic” random variables such that $ρ (X, Y) = a$ and $ρ (Y, Z) = b$ , we should on average expect that $ρ (X, Z) = a b$ .

The connection of this geometric claim to correlations is straightforward once we know that correlations can be interpreted as inner products. Indeed, if we consider real-valued random variables on a finite probability space with $n$ elements, then we can identify the space of random variables with $R^{n}$ , the space of random variables of mean zero with $R^{n - 1}$ and the space of random variables with mean zero and of unit variance with the unit sphere $S^{n - 2}$ . The appropriate notion of considering a “generic” or “average” case here is to make our setup rotationally invariant, and that’s exactly what the uniform distribution on the unit sphere does.

Now that we understand the connection to correlations, let’s see how to prove this claim.

Proof

As with the proof of the first claim, we take the familiar orthogonal projections

$X = a Y + \sqrt{1 - a^{2}} H_{X Y} Z = b Y + \sqrt{1 - b^{2}} H_{Z Y}$

and write the inner product we care about as

$⟨ X, Z ⟩ = a b + \sqrt{(1 - a^{2}) (1 - b^{2})} ⟨ H_{X Y}, H_{Z Y} ⟩$

Now, let’s think about what the joint distribution of $H_{X Y}$ and $H_{Z Y}$ is like. These vectors could be anything orthogonal to $Y$ , and the vectors orthogonal to $Y$ are located on an “equator” or “great circle” of $S^{n}$ which is isomorphic to $S^{n - 1}$ . As the measure we’re taking expectations over on $S^{n}$ is rotationally invariant, we notice that the individual distributions of $H_{X Y}, H_{Z Y}$ must be uniform on the great circle. Furthermore, as we can rotate $X$ and $Z$ independently without affecting the conditioning, we also deduce that their distributions are independent conditional on $Y$ .

Once we know all of this, it’s clear that we must have $E [⟨ H_{X Y}, H_{Z Y} ⟩ | Y, ⟨ X, Y ⟩ = a, ⟨ Y, Z ⟩ = b] = 0$ because of reflection symmetry: for any pair $(H_{X Y}, H_{Z Y})$ with positive inner product, we can simply reflect $H_{X Y}$ across the origin while staying on the great circle orthogonal to $Y$ and find a pair with an inner product having equal magnitude but opposite sign, so the expectation must vanish.

Then, defining $C = (⟨ X, Y ⟩ = a \land ⟨ Y, Z ⟩ = b)$ for ease of notation, Adam’s law finishes the proof:

$E [⟨ X, Z ⟩ | C] = a b + \sqrt{(1 - a^{2}) (1 - b^{2})} \cdot E [⟨ H_{X Y}, H_{Z Y} ⟩ | C] = a b + \sqrt{(1 - a^{2}) (1 - b^{2})} \cdot E [E [⟨ H_{X Y}, H_{Z Y} ⟩ | C, Y] | C] = a b$

Interpretation

The above statement is essentially a formalization of the Bayesian claim that “when $X, Y$ are positively correlated and $Y, Z$ are positively correlated, it’s likely that $X, Z$ are positively correlated as well”. So while the properties for correlation to be strictly transitive are rather stringent as laid out in Claim 1, a weaker notion of “transitivity in expectation” holds for arbitrarily weak positive correlations, as $a b$ is strictly positive whenever both $a, b$ are.

However, the statement is also a negative result in the sense that it implies a quadratic decay in the strength of the correlation we should expect. If $ρ (X, Y) = ρ (Y, Z) = a$ and $X, Y, Z$ are otherwise generic random variables, we should only expect $ρ (X, Z) = a^{2}$ . Because of this second-order scaling, relationships which are quite strong may become weak in expectation once we go through a “transitivity link”. For instance, a correlation of $0.3$ is considered quite strong in the social sciences where everything is noisy, but a correlation of ${0.3}^{2} = 0.09$ is not.

Why write this post?

Correlations come up in many contexts, but most people don’t have very good intuition for how correlation behaves in general. I see arguments of the form “A is correlated with B and B is correlated with C, so A should correlate with C” or “it’s not surprising that A would correlate with C” quite often. However, it’s important to have quantitative intuitions for what to expect here. For example, if $ρ (A, B) = ρ (B, C) = 0.5$ and $ρ (A, C) = 0.4$ , that suggests $A, C$ are more strongly correlated than we would expect just on the basis of the connection between them that goes through $B$ : this would be some evidence for a latent factor that all three variables have some kind of loading on.

I hope that this post helps clear up some confusion about the properties of correlations involving three or more variables and what we can and can’t expect from correlation in such contexts.