Trade sizes fall into 4 buckets with true probabilities P of 1/2, 1/4, 1/8 and 1/8. A model Q treats the buckets as equally likely. What is D_KL(P ∥ Q) in bits? Give a fraction.

Learn / Library / Statistics

A flat model of a skewed law

StatisticsEasy

Trade sizes fall into $4$ buckets with true probabilities $P$ of $\frac{1}{2}$ , $\frac{1}{4}$ , $\frac{1}{8}$ and $\frac{1}{8}$ . A model $Q$ treats the buckets as equally likely. What is $D_{KL} (P ∥ Q)$ in bits? Give a fraction.

Checking an answer needs JavaScript. The hint and the solution open without it.

Hint

Use $D_{KL} (P ∥ Q) = H (P, Q) - H (P)$ .

Solution

Under $Q$ each bucket costs $2$ bits, so the cross-entropy is $2$ .
The entropy of $P$ is $\frac{1}{2} \times 1 + \frac{1}{4} \times 2 + \frac{1}{8} \times 3 + \frac{1}{8} \times 3 = \frac{7}{4}$ .
The divergence is $2 - \frac{7}{4} = \frac{1}{4}$ .

Answer: $\frac{1}{4}$

Next question Back to the library