Miss_Figg

Karma: 2

Miss_Figg 8 Jul 2019 15:39 UTC
3 points
on: Is the sum individual informativeness of two independent variables no more than their joint informativeness?
We want to prove:
$\int \int p_{x s} (x, s) log \frac{p_{x s} (x, s)}{p_{s} (s) p_{x} (x)} d x d s + \int \int p_{y s} (y, s) log \frac{p_{y s} (y, s)}{p_{s} (s) p_{y} (y)} d y d s \leq \int \int \int p_{x y s} (x, y, s) log \frac{p_{x y s} (x, y, s)}{p_{s} (s) p_{x y} (x, y)} d x d y d s$
This can be rewritten as:

$\int \int \int p_{x y s} (x, y, s) log \frac{p_{s x} (s, x)}{p_{s} (s) p_{x} (x)} d x d y d s + \int \int \int p_{x y s} (x, y, s) log \frac{p_{s y} (s, y)}{p_{s} (s) p_{y} (y)} d x d y d s \leq \int \int \int p_{x y s} (x, y, s) log \frac{p_{x y s} (x, y, s)}{p_{s} (s) p_{x} (x) p_{y} (y)} d x d y d s$
After moving everything to the right hand side and simplifying, we get:

$\int \int \int p_{x y s} (x, y, s) log \frac{p_{x y s} (x, y, s) p_{s} (s)}{p_{x s} (x, s) p_{y s} (y, s)} d x d y d s \geq 0$
Now if we just prove that $q (x, y, s) = \frac{p_{x s} (x, s) p_{y s} (y, s)}{p_{s} (s)}$ is a probability distribution, then the left hand side is $K L (p_{x y s} (x, y, s), q)$ , and Kullback-Leibler divergence is always nonnegative.
Ok, q is obviously nonnegative, and its integral equals 1:
$\int \int \int \frac{p_{x s} (x, s) p_{y s} (y, s)}{p_{s} (s)} d x d y d s = \int \frac{p_{s} (s) p_{s} (s)}{p_{s} (s)} d s = \int p_{s} (s) d s = 1$
Q.e.d.