A parameter θ takes 4 values with joint probabilities p(θ, D) equal to 1/8, 1/16, 1/32, 1/32. An approximation is q = (1/2, 1/8, 1/8, 1/4). Compute the evidence lower bound Σ q log₂(p(θ, D)/q) in bits. Answer as a fraction in lowest terms, with its sign.

Learn / Library / Statistics

An ELBO from the joint

StatisticsHard

A parameter $θ$ takes $4$ values with joint probabilities $p (θ, D)$ equal to $\frac{1}{8}$ , $\frac{1}{16}$ , $\frac{1}{32}$ , $\frac{1}{32}$ . An approximation is $q = (\frac{1}{2}, \frac{1}{8}, \frac{1}{8}, \frac{1}{4})$ . Compute the evidence lower bound $\sum q log_{2} (\frac{p ( θ , D )}{q})$ in bits. Answer as a fraction in lowest terms, with its sign.

Checking an answer needs JavaScript. The hint and the solution open without it.

Hint

Divide each joint probability by its $q$ , take $log_{2}$ , and weight by $q$ .

Solution

The ratios of joint to $q$ are $\frac{1}{4}$ , $\frac{1}{2}$ , $\frac{1}{4}$ , $\frac{1}{8}$ , with logs $- 2$ , $- 1$ , $- 2$ , $- 3$ .
Weighting by $q$ gives $- 1 - \frac{1}{8} - \frac{1}{4} - \frac{3}{4} = - \frac{17}{8}$ .
So the bound is $- \frac{17}{8}$ bits, and the answer is $- \frac{17}{8}$ .

Answer: $- \frac{17}{8}$

Next question Back to the library