On the same 5 outcomes in order, P has probabilities 1/2, 1/4, 1/8, 1/16, 1/16 and Q has 1/4, 1/16, 1/2, 1/16, 1/8. Find D_KL(Q ∥ P) − D_KL(P ∥ Q) in bits. Give a fraction.

Learn / Library / Statistics

Both directions

StatisticsHard

On the same $5$ outcomes in order, $P$ has probabilities $\frac{1}{2}$ , $\frac{1}{4}$ , $\frac{1}{8}$ , $\frac{1}{16}$ , $\frac{1}{16}$ and $Q$ has $\frac{1}{4}$ , $\frac{1}{16}$ , $\frac{1}{2}$ , $\frac{1}{16}$ , $\frac{1}{8}$ . Find $D_{KL} (Q ∥ P) - D_{KL} (P ∥ Q)$ in bits. Give a fraction.

Checking an answer needs JavaScript. The hint and the solution open without it.

Hint

List $log_{2} (\frac{p}{q})$ for each outcome first.

Solution

The values of $log_{2} (\frac{p}{q})$ are $1$ , $2$ , $- 2$ , $0$ and $- 1$ .
With weights $P$ , $D_{KL} (P ∥ Q) = \frac{1}{2} \times 1 + \frac{1}{4} \times 2 - \frac{1}{8} \times 2 - \frac{1}{16} \times 1 = \frac{11}{16}$ .
With weights $Q$ and signs flipped, $D_{KL} (Q ∥ P) = - \frac{1}{4} \times 1 - \frac{1}{16} \times 2 + \frac{1}{2} \times 2 + \frac{1}{8} \times 1 = \frac{3}{4}$ .
The difference is $\frac{3}{4} - \frac{11}{16} = \frac{1}{16}$ .

Answer: $\frac{1}{16}$

Next question Back to the library