Lorxus comments on Natural Latents: The Math

[Lorxus]

Probabilities of zero are extremely load-bearing for natural latents in the exact case...

Dumb question: Can you sketch out an argument for why this is the case and/​or why this has to be the case? I agree that ideally/​morally this should be true, but if we’re already accepting a bounded degree of error elsewhere, what explodes if we accept it here?

[johnswentworth]

Consider the exact version of the redundancy condition for latent $\Lambda$ over $X_1,X_2$:

$P[\Lambda ,X_1,X_2]=P\left[\Lambda |X_1\right]P[X_1,X_2]$

and

$P[\Lambda ,X_1,X_2]=P\left[\Lambda |X_2\right]P[X_1,X_2]$

Combine these two and we get, for all $\Lambda ,X_1,X_2$:

$P\left[\Lambda |X_1\right]=P\left[\Lambda |X_2\right]$ OR $P[X_1,X_2]=0$

That’s the foundation for a conceptually-simple method for finding the exact natural latent (if one exists) given a distribution $P[X_1,X_2]$:

• Pick a value $X_1,X_2$ which has nonzero probability, and initialize a set $S$ containing that value. Then we must have $P[\Lambda |X\in S]=P\left[\Lambda |X_1\right]=P\left[\Lambda |X_2\right]$ for all $\Lambda$.

• Loop: add to $S$ a new value $X_1^{\prime },X_2$ or $X_1,X_2^{\prime }$ where the value $X_2$ or $X_1$ (respectively) already appears in one of the pairs in $S$. Then $P\left[\Lambda |X_1^{\prime }\right]=P[\Lambda |X\in S]$ or $P\left[\Lambda |X_2^{\prime }\right]=P[\Lambda |X\in S]$, respectively. Repeat until there are no more candidate values to add to $S$.

• Pick a new pair and repeat with a new set, until all values of $X$ have been added to a set.

• Now take the latent to be the equivalence class in which $X$ falls.

Does that make sense?

[Lorxus]

As I also said in person, very much so!