The Honest Backtest Checklist

A backtest can be made to show almost anything. Before you trust one — or pay for someone else's "proven" system — run it through this checklist.

A backtest can be made to show almost anything. Before you risk a cent — or pay for someone else's "proven" system — run it through this checklist. If you can't answer yes to most of these, the edge you're looking at is probably noise, not skill.

1. Data & sample

• Out-of-sample test. Was part of the data held back and never used for tuning? A strategy that only shines on the data it was optimised on tells you nothing.
• Enough trades. Are there hundreds of trades, not a dozen? A great-looking curve from 15 trades is a coin flip, not an edge.
• Multiple regimes. Was it tested across bull, bear and sideways markets — not just one lucky trend?
• Clean data. No survivorship bias (delisted tickers included), no lookahead (only information available at the time)?

2. Method

• Parameter count. How many settings were tuned? Every extra parameter fits more noise. Fewer is safer.
• Number of variants tried. Were 5 combinations tested — or 10,000? If you try enough, one will look spectacular by pure chance (data-mining).
• Walk-forward. Does it survive re-optimising on a rolling window and trading the next, unseen window? This is the closest thing to live conditions.
• A plausible reason. Is there an economic or structural reason it should work — or is it a pattern with no explanation?

3. Statistics

• Deflated Sharpe Ratio. Has the Sharpe been corrected for how many strategies were tried? A raw Sharpe of 2.0 can deflate to nothing once you account for the search.
• Probability of Backtest Overfitting (PBO). Is there any estimate of how likely the "best" configuration is to underperform out-of-sample?
• Drawdown you can actually stomach. Is the maximum historical drawdown known — and could you sit through it without abandoning the system?

4. Costs & reality

• Costs included. Do the results include realistic spreads, commissions and slippage? Most small edges vanish once costs are subtracted.
• Live or forward record. Is there any track record on data that came after the backtest was built — ideally a real, documented one?

The honest test: cross out every line you can't answer with a confident yes. What's left is how much of that beautiful equity curve you can actually trust.

Doing this by hand is slow. quantcheck runs the statistical parts for you — multiple-testing correction, Deflated Sharpe and an overfitting verdict — for free, no signup for the result. Upload a backtest and find out in minutes whether the edge is real or just a well-dressed random number.