(No title) (answer)

Joint entropy of two random variables

For two ensembles, $X = (x, A_{x}, P_{x})$ and $Y = (y, A_{y}, P_{y})$ , where there may be dependency between $P_{x}$ and $P_{y}$ , the joint entropy of $X$ , $Y$ is:

H (X, Y) = \sum_{x \in A_{x}} \sum_{y \in A_{y}} P (x, y) l o g \frac{1}{P (x, y)}

Entropy is additive for independent random variables.

Proof

\begin{aligned} H (X, Y) & = \sum_{x \in A_{x}} \sum_{y \in A_{y}} P (x) P (y) l o g \frac{1}{P (x) P (y)} \\ = \sum_{x \in A_{x}} \sum_{y \in A_{y}} P (x) P (y) l o g \frac{1}{P (x)} + \sum_{x \in A_{x}} \sum_{y \in A_{y}} P (x) P (y) l o g \frac{1}{P (y)} \\ = \sum_{x \in A_{x}} P (x) l o g \frac{1}{P (x)} + \sum_{y \in A_{y}} P (y) l o g \frac{1}{P (y)} (the first sum's terms are independent of y, and the second's independent of x) \\ = H (X) + H (Y) \end{aligned}

07.05.2019