Backtesting forecasts

Idea: Ask about the calendar

A backtest is a claim about the past as it was lived. Use data in the order the world produced it.

• When was each forecast made?
• What was known at that moment?

Example: Walk forward, never shuffle

$1000$ days, a training window of $500$, test blocks of $20$. How many refits?

1. Fit, forecast the next block, roll both forward
2. $\frac{1000-500}{20}=25$

Formula: Error sizes

$\mathrm{RMSE}=\sqrt{\mathrm{mean}of(y-\hat{y}{)}^2}\ge \mathrm{MAE}$

$\mathrm{MAE}$ averages the size of a miss. $\mathrm{RMSE}$ punishes big misses more and is never below $\mathrm{MAE}$.

Quick check: Compute the $\mathrm{RMSE}$

Misses of $1$, $-1$, $3$, $-3$ and $5$. What is the $\mathrm{RMSE}$?

Example: Check an interval

An $80\%$ interval holds on $162$ of $225$ days. How many standard deviations short is that?

1. Expected $225\times 80\%=180$, variance $225\times 0.8\times 0.2=36$
2. Shortfall $180-162=18$, $\frac{18}{6}=3$

Idea: Costs turn signals into strategies

A forecast is worth only what survives trading it.

• $8$ bp gross, half the book traded at $6$ bp: $8-0.5\times 6=5$
• Double the turnover: $8-1\times 6=2$
• A signal from the close cannot trade at that close

Idea: 3 ways to fool yourself

Leakage, survivorship and repeated tuning all make a backtest look better than it is.

• Look-ahead: revised data, full-sample scaling, period-end dates
• Survivorship: use membership lists as they stood on each date
• Tuning: $20$ tries at $5\%$ give $20\times 5\%=1$ false find

Common mistake: Hiding the trial count

The best of many tries always looks good: $1-0.95^{20}\approx 0.64$ for at least one false find.

Instead Report how many variants you tried, adjust for them, and touch the final holdout once.

Idea: Say it in the room

Name the split first, then the metric, then the costs.

• Ask how many variants were tried
• Ask whether the universe and the data are as they stood on each date