Maximum likelihood estimation

Idea: The model read backwards

Fix the data you saw and read $p(x\mid \theta )$ as a function of $\theta$. That is the likelihood $L(\theta )$.

• It is not a probability of $\theta$
• It need not add up to $1$ across $\theta$
• The estimate $\hat{\theta }$ makes your data most probable

Idea: The recipe

Independent data multiply, so take logs: the sum $\ell (\theta )$ peaks at the same $\theta$.

• Write $\ell (\theta )$ and differentiate
• Set the slope to zero
• Check that the second derivative is negative

Formula: Log likelihood

$\ell (\theta )=\sum \log p(x_i\mid \theta )$

Add the log chance of each observation. The peak of this sum is the estimate.

Example: The pooled order rate

Orders arrive at a steady random rate: $4$ in a $10$ minute window, $8$ in a $30$ minute one. Find the maximum likelihood rate.

1. $\ell (\lambda )=\sum (k_i\log \lambda -\lambda t_i)$
2. The slope is zero at $\hat{\lambda }=\sum k_i/\sum t_i$
3. $\hat{\lambda }=\frac{4+8}{10+30}=\frac{3}{10}$ per minute

Idea: Pool, do not average rates

Averaging the window rates gives $\frac{1}{3}$: a short window gets the same vote as a long one.

• A longer window carries more evidence
• Check: $\frac{3}{10}\times 40=12$ orders, as observed

Quick check: A variance estimate

Daily returns $2$, $4$ and $9$ come from a normal law. What is the maximum likelihood variance ${\hat{\sigma }}^2$?

Common mistake: Expecting no bias

The variance estimate divides by $n$ and runs low: $E[{\hat{\sigma }}^2]=\frac{(n-1){\sigma }^2}{n}$.

Instead For an unbiased one divide by $n-1$: $\frac{26}{2}=13$ instead of $\frac{26}{3}$.

Formula: Invariance

$\text{estimate}ofg(\theta )=g(\hat{\theta })$

Transform the estimate, not the derivation: $6$ wins of $8$ give $\hat{p}=\frac{3}{4}$ and odds $3$.

Common mistake: Trusting large-sample promises

Consistency and a normal limit need a right model, a data range free of $\theta$ and $\theta$ off the edge.

Instead Under $\mathrm{Uniform}(0,\theta )$ the estimate is $\max x_i$: no zero slope, and it always runs low.

Idea: Say it in the room

Write the likelihood first: it forces you to state the model.

• Logs, slope, second derivative, then numbers
• Finish with a units or extreme-case check