Base rates

Idea: A good detector, a rare event

A signal can be right on most cases and still be wrong most of the times it fires.

• The catch rate is measured among true cases
• You need the share of flags that are true

Example: The mistyped-order flag

$1\%$ of trades are mistakes. A model flags $90\%$ of mistakes and $5\%$ of correct trades. A trade is flagged. Chance it is a mistake?

1. $2000$ trades: $20$ mistakes, $1980$ correct
2. Flags: $20\times 90\%=18$ true, $1980\times 5\%=99$ false
3. Chance: $\frac{18}{117}=\frac{2}{13}$

Formula: Base rate, hit rate, false alarms

$P(H\mid E)=p\times \frac{s}{p\times s+(1-p)\times f}$

p is the base rate, s the share of true cases caught, f the false positive rate.

Idea: The base rate decides

The same model on crowds of $2000$ trades:

• Base rate $1\%$: $\frac{18}{18+99}=\frac{2}{13}$
• $10\%$: $\frac{180}{180+90}=\frac{2}{3}$
• $50\%$: $\frac{900}{900+50}=\frac{18}{19}$

Idea: False alarms are the lever

For a rare event, cutting false alarms helps more than catching everything.

• Catch every mistake: $\frac{20}{20+99}\approx 0.17$
• Halve the false alarms: $\frac{18}{18+49.5}=\frac{4}{15}$

Common mistake: Calling it 90% accurate

$90\%$ is measured among mistakes, not among flags.

Instead Ask for the share of flags that are true. A model that never flags is right on $99\%$ and finds nothing.

Quick check: A second, independent check

The flagged trades go through a second, independent check with the same rates. Chance a twice-flagged trade is a mistake?

Idea: The posterior becomes the prior

The flagged trades are a new crowd, already richer in mistakes.

• The second check must be independent given the true state
• Running the same model again repeats its own mistake

Idea: What to say in the room

Ask for the base rate first; without it there is no answer.

• Say true and false flags as a pair of counts, then divide
• To improve a rare-event test, cut false alarms or add a second check