Variational inference

Idea: A posterior you cannot afford

The evidence $p(D)$ sums over every $\theta$ and cannot be done; sampling all night is too slow.

• Variational inference turns the integral into an optimisation
• Pick a simple family, find the member $q$ closest to the posterior

Formula: KL divergence

$D_{\mathrm{KL}}(q\parallel p)=\sum q{\log }_2\left(\frac{q}{p}\right)$

Zero only when q matches p. Not symmetric: this one averages over q.

Formula: The evidence lower bound

${\log }_{2}p(D)=\text{ELBO}(q)+D_{\mathrm{KL}}(q\parallel \text{posterior})$

Raise the bound and the divergence falls as much. The bound needs only prior times likelihood.

Example: The gap by hand

Joint $\frac{1}{4}$, $\frac{1}{8}$, $\frac{1}{8}$ over $3$ values of $\theta$, and $q=\left(\frac{1}{4},\frac{1}{4},\frac{1}{2}\right)$. How many bits is the gap to the evidence?

1. Evidence: $\frac{1}{4}+\frac{1}{8}+\frac{1}{8}=\frac{1}{2}$, its log is $-1$
2. Bound: $0-\frac{1}{4}-1=-\frac{5}{4}$
3. Gap: $-1+\frac{5}{4}=\frac{1}{4}$

Idea: The price of independence

Mean-field $q$ treats each parameter as independent: fast, with a known bias.

• It gets the mean right and the variance too small
• Its divergence punishes mass where p is thin, not parts of p it misses

Quick check: How narrow is it?

A normal posterior has $\mathrm{Var}(x)=\mathrm{Var}(y)=4$ and $\mathrm{Cov}(x,y)=3$. What variance does the mean-field $q$ give $x$?

Common mistake: Trusting a variational error bar

It is often too narrow, and most confident when correlation is strongest.

Instead Check against a short sampling run before sizing. Compare bounds only within one model.

Idea: Where it fits

Monte Carlo is slow but gets there. Variational inference is fast, and its bias stays.

• Use it for a posterior on a deadline
• Remember which way it leans: too confident