Mutual information

Idea: Zero correlation, full information

Calm days return $1$ or $-1$, wild days $4$ or $-4$. The correlation with the regime is zero.

• Yet one look at the return reveals the regime
• Correlation sees only straight-line links

Formula: Mutual information

$I(X;Y)=H(Y)-H(Y\mid X)$

The drop in doubt about Y once you know X. Zero exactly when X and Y are independent.

Formula: The fastest form by hand

$I(X;Y)=H(X)+H(Y)-H(X,Y)$

Symmetric in X and Y. It also equals $D_{\mathrm{KL}}(P_{XY}\parallel P_X{P}_Y)$.

Example: A signal that is often silent

A move $X$ is up or down with $\frac{1}{2}$. On half the days a signal reports it exactly; otherwise it is silent. Find $I(X;Y)$.

1. $H(X)=1$ and $H(Y)=\frac{1}{4}\times 2+\frac{1}{4}\times 2+\frac{1}{2}=\frac{3}{2}$
2. $4$ equal cells of $\frac{1}{4}$: $H(X,Y)=2$
3. $I(X;Y)=1+\frac{3}{2}-2=\frac{1}{2}$

Idea: Half a bit makes sense

Half the days reveal the full $1$ bit, the rest nothing: $\frac{1}{2}\times 1+\frac{1}{2}\times 0=\frac{1}{2}$.

• A feed that loses a share of messages loses that share of $H(X)$

Quick check: The regime from a return

Each regime has chance $\frac{1}{2}$, each sign $\frac{1}{2}$, and the size of the return gives the regime away. What is $I(R;V)$ in bits?

Idea: Hard to estimate

$10$ bins per series make $100$ cells; $100$ days leave about $1$ day per cell.

• The plug-in estimate is biased upward
• Shuffle one series and rerun the same estimator

Common mistake: Over-reading the numbers

Zero correlation does not mean independence, and a measured value above $0$ does not prove dependence.

Instead On a finite sample the estimate is almost never exactly $0$: compare it with shuffled data.

Idea: Say it in an interview

The drop in entropy of one variable once you know the other.

• Symmetric
• Catches nonlinear links
• Hard to estimate from short samples