Fisher information and the Cramér-Rao bound

Idea: How sharply the data point

The score $S(\theta )$ is the slope of the log likelihood. It averages zero; its swing is the information.

• A steep likelihood locates $\theta$ well
• A flat one does not
• Information adds over independent data: $I_n=n\times I$

Formula: Fisher information

$I(\theta )=E[S^2]=\mathrm{Var}(S)$

The variance of the score, and also minus the average curvature of the log likelihood.

Formula: The Cramér-Rao bound

$\mathrm{Var}(T)\ge \frac{1}{I_n}(\theta )$

No unbiased estimator of θ in a smooth model has a smaller variance than this.

Example: How precise can a fill rate be

Orders fill with chance $p=\frac{1}{5}$. What is the smallest standard error an unbiased estimate can have from $100$ orders?

1. One order: $I(p)=\frac{1}{p(1-p)}$
2. $\frac{1}{\frac{1}{5}\times \frac{4}{5}}=\frac{25}{4}$, so $100\times \frac{25}{4}=625$
3. Variance $\frac{1}{625}=0.0016$, error $0.04$

Quick check: Orders for a 1 point error

Same fill rate $p=\frac{1}{5}$. How many orders does the bound need for a standard error of $0.01$?

Idea: Efficiency

Efficiency is the bound over an estimator’s variance: the share of the information it uses.

• Reaching the bound means efficient
• The sample fill rate $\hat{p}$ reaches it exactly
• Ticks up, down or flat: up ticks alone use $\frac{1}{3}$

Quick check: Information in a tick

A tick goes up with chance $\theta$, down with chance $\theta$, flat otherwise. What is $I(\theta )$ at $\theta =\frac{1}{4}$?

Common mistake: The bound as a floor for everything

It covers unbiased estimators in smooth models only. The constant $T=\frac{1}{2}$ has zero variance.

Instead Biased shrinkage can sit under it, and $\mathrm{Uniform}(0,\theta )$ beats the $\frac{1}{n}$ rate. Use $I_n$, not $I$.

Idea: Say it in the room

Information is the variance of the score, and it adds across independent data.

• One observation, times $n$, inverted: the best variance
• Beats the bound? Ask if it is biased or irregular