> ## Content Index > Fetch the complete content index at: https://quantdojo.ai/llms.txt > Use this file to discover other available public pages before exploring further. # How to Read a Strategy Vendor's Backtest Without Getting Fooled - URL: https://quantdojo.ai/strategy-vendor-backtest-red-flags-checklist/ - Published: 2026-08-29T08:05:43.000Z - Updated: 2026-08-29T08:05:43.000Z - Description: Most strategy vendors sell you the backtest they want you to see. Here's the systematic checklist that exposes what they're hiding. - Author: Wolfgang Lämmle - Tags: backtesting, due diligence, strategy evaluation, overfitting, validity A vendor sends you a backtest. The equity curve climbs at a 45-degree angle. The Sharpe is 2.8\. The maximum drawdown is a tidy 7%. Your first instinct is excitement. Your second instinct — the one you should trust — is skepticism. Strategy vendors are not your research partners. They are selling a product, and the backtest is their marketing material. Most of what looks impressive in a third-party backtest has been — consciously or not — engineered to look that way. Your job as a buyer is to reverse-engineer it back to reality. This guide gives you a systematic checklist to do exactly that. Work through it in order. If a vendor's backtest fails more than two items, walk away. --- ## Why Third-Party Backtests Are Almost Always Optimistic Every strategy developer — even an honest one — runs dozens of iterations before arriving at the version they publish. Each iteration is a silent hypothesis test. By the time a strategy looks good enough to sell, it has been unconsciously fitted to the historical data it was tested on. The result: the published backtest represents the *best* of many tries, not a single, pre-specified test. That selection process inflates every metric. Sharpe looks better than it should. Drawdown looks smaller. Win rate looks higher. The equity curve looks smoother. None of this requires deliberate fraud — it's just what happens when humans optimize until they're happy, then stop. We tested this directly: [generating 1,500 strategies produced 114 that "passed" standard criteria on Bitcoin — and none of them were real edges](https://quantdojo.ai/1500-strategies-114-passed-none-real/). Vendors are running the same process, then handing you the one survivor. --- ## The Vendor Backtest Checklist ### 1\. Count the Trades **Minimum acceptable: 100+ trades across the full backtest period.** A strategy with 40 trades is statistically meaningless. You cannot distinguish edge from luck. Vendors often show shorter histories because longer histories either don't exist or don't look as good. If the trade count is low, no other metric matters — the sample size is too small to interpret. Also check for clustering: if 60% of trades occurred in one six-month bull run, the strategy may have one regime baked in, not a generalizable edge. ### 2\. Measure the In-Sample Period **Red flag: backtests shorter than 5 years, or with no explicit in-sample vs. out-of-sample split.** The longer the history, the more market regimes it covers — trending, mean-reverting, high-volatility, low-volatility, crash events. A 12-month backtest on crypto in 2021 tells you nothing about the strategy's robustness. Ask the vendor what the in-sample period is and what it looked like. If they can't answer, that's your answer. No out-of-sample period at all is an immediate disqualifier. Walk-forward validation — where parameters are re-optimized periodically and tested on unseen data — is the minimum acceptable standard. If you're unfamiliar with why, [walk-forward analysis is the right way to validate a trading strategy](https://quantdojo.ai/walk-forward-analysis-validate-trading-strategy/) and it's not optional. ### 3\. Audit the Parameter Count **Rule of thumb: no more than 1 free parameter per 100 trades.** Every adjustable parameter is an opportunity to fit the historical data. A strategy with 8 parameters and 150 trades has almost certainly been over-fitted — there's enough degrees of freedom to carve out a beautiful equity curve from pure noise. Ask the vendor how many parameters were optimized and over what range. A legitimate vendor knows this number immediately. The [curve-fitting trap](https://quantdojo.ai/curve-fitting-trap-optimized-strategy-fails-live/) is why optimized strategies so often die the moment they go live. More parameters means a more precise map of the past — and a less useful guide to the future. ### 4\. Reconstruct the Transaction Costs **Non-negotiable: realistic commissions, spread, and slippage must be deducted.** This is where vendor backtests most commonly lie — not by intent, but by omission. Zero-slippage assumptions are endemic. Spread modeled as a fixed tight number is common. For liquid instruments it might look minor; for crypto, small-caps, or any illiquid asset it can be the difference between a 2.0 Sharpe and a 0.4. Ask for the exact cost assumptions used. Then verify them against your actual broker conditions. [Slippage alone is quietly killing most backtest edges](https://quantdojo.ai/slippage-backtest-model-execution-costs/) — and vendors have every incentive to minimize it in their presentation. If the backtest uses market orders on illiquid instruments with zero slippage, the results are fiction. ### 5\. Interrogate the Sharpe Ratio **A raw Sharpe of 2.0 means very little. Ask for the Deflated Sharpe.** The raw Sharpe ratio does not account for the number of strategies tested, the length of the backtest, or the non-normality of returns. After multiple testing — which every vendor has done — a Sharpe of 2.0 could easily be noise with a 60%+ probability of not being a real edge. The Deflated Sharpe Ratio adjusts for all of these factors. If the vendor hasn't heard of it, that tells you something. If they have and won't provide it, that tells you more. Our breakdown of [why your strategy's Sharpe of 2.1 is probably noise](https://quantdojo.ai/deflated-sharpe-ratio-multiple-testing-noise/) explains the math and what thresholds to actually care about. ### 6\. Look for Out-of-Sample Results **If there's no genuine holdout period, the backtest proves nothing.** Out-of-sample data is data the strategy developer never touched during development. It's the only honest test of generalization. Look for: - **An explicit holdout period** — ideally the most recent data, since that's what the strategy will trade next. - **Walk-forward segments** — multiple in-sample/out-of-sample windows rolled forward in time. - **Paper trading or live results** — even a short live track record is worth far more than years of backtest. If a vendor can only show in-sample results, they are asking you to fund their out-of-sample test. Don't. ### 7\. Check the Overfitting Probability Directly **Tools exist. Use them.** Probability of Backtest Overfitting (PBO) — derived from the Combinatorially Symmetric Cross-Validation framework — gives you a quantitative estimate of the probability that a strategy's performance is a product of overfitting rather than genuine edge. A PBO above 50% is a failing grade. Above 70% is a red flag in isolation. You don't need to run this yourself. [QuantCheck — our free no-code backtest validation tool](https://quantdojo.ai/quantcheck/) — runs this audit for you. Drop in the backtest stats and get a verdict. If the vendor's strategy fails there, you have a number to show them, not just a feeling. For a deeper look at what PBO actually measures and how to interpret it, [understanding the probability of backtest overfitting](https://quantdojo.ai/probability-backtest-overfitting-pbo-cscv/) is worth the read before any purchase. --- ## The Presentation Tricks That Should Trigger Instant Skepticism Beyond the core checklist, watch for these specific presentation choices that are almost always deliberate: - **Percentage returns instead of risk-adjusted metrics.** "Up 300%" sounds great. At what drawdown? With what volatility? A raw return number with no context is meaningless. - **Cherry-picked date ranges.** Does the equity curve start right after a bad period? Does it end before a drawdown? If the vendor chose the dates, they chose the narrative. - **No drawdown chart shown.** Maximum drawdown is easy to hide if you only show cumulative returns. Ask for both. - **Win rate as the headline metric.** A 90% win rate is entirely consistent with a losing strategy — [the win-rate trap is real and common](https://quantdojo.ai/a-90-win-rate-that-loses-money-the-win-rate-trap/) in vendor marketing. - **Survivorship in the asset selection.** Testing only on assets that performed well during the backtest period is a subtle but devastating form of data snooping. --- ## What a Legitimate Vendor Backtest Looks Like Honest vendors exist. They're rare, but they can be identified. A trustworthy third-party backtest includes: - **Full parameter disclosure** — what was optimized, over what range, and when. - **Explicit cost assumptions** — commission schedule, spread model, slippage estimate, and the rationale for each. - **Out-of-sample or walk-forward results** — not just shown, but the in-sample period clearly labeled separately. - **Risk-adjusted metrics** — Sharpe, Sortino, Calmar ratio, and maximum drawdown — not just returns. - **A Monte Carlo stress test** — showing the distribution of outcomes under randomized trade ordering or resampled returns, not just the single historical path. - **Honest limitations** — what market conditions the strategy does not work in, what its known failure modes are. If a vendor provides all of this unprompted, that is a strong positive signal. If they push back on any of it, walk. --- ## Your Pre-Purchase Due Diligence: The Quick Reference Before spending money on any third-party strategy, run through this list: - **Trade count** — 100+ minimum; check for regime clustering. - **Backtest length** — 5+ years; covers multiple market regimes. - **Parameter count** — roughly 1 per 100 trades maximum. - **Transaction costs** — realistic commission, spread, and slippage explicitly stated. - **Sharpe quality** — Deflated Sharpe or equivalent multiple-testing adjustment provided. - **Out-of-sample evidence** — genuine holdout or walk-forward results, not in-sample only. - **Overfitting audit** — PBO calculated and acceptable; use [QuantCheck](https://quantdojo.ai/quantcheck/) to verify independently. - **Presentation honesty** — drawdown shown, date range selection explained, no win-rate-only framing. For your own strategies, the [Honest Backtest Checklist](https://quantdojo.ai/honest-backtest-checklist/) applies the same rigor in the other direction — so you know what a vendor *should* have done before publishing. --- ## Frequently Asked Questions ### What is the most common red flag in a vendor backtest? Missing or unrealistic transaction costs — particularly slippage — are the single most common issue. A backtest that assumes zero slippage on market orders, or a fixed-tight spread on an illiquid instrument, can show a profitable strategy that loses money in live trading from execution costs alone. Always ask for the exact cost model used and compare it to your actual broker conditions. ### How many trades does a backtest need to be statistically meaningful? A commonly cited minimum is 100 trades, though more is always better. Below 100, the confidence intervals around any performance metric are wide enough to make the result interpretable as pure luck. If a vendor's strategy only generated 40 or 50 trades across its entire history, there is no statistical basis for trusting the reported Sharpe or win rate. ### Can I validate a vendor's backtest myself without coding skills? Yes. [QuantCheck](https://quantdojo.ai/quantcheck/) is a free, no-code tool that runs a systematic overfitting audit — including a Probability of Backtest Overfitting estimate — from the backtest summary statistics alone. You don't need access to the underlying code or data. Enter the reported metrics, and QuantCheck returns an independent verdict on whether the results are plausibly real or statistically suspect. --- ## The Honest Bottom Line Every vendor backtest is, at minimum, a sales document. That doesn't make the vendor dishonest — it makes the incentives misaligned. Your job is to treat every reported metric with methodical skepticism, not cynicism. Work the checklist. Ask for what isn't shown. Use independent tools to audit what you're given. The equity curve a vendor shows you is the best version of the story. Your job is to find the rest of it. *Nothing here is financial advice — just a checklist to help you buy a real edge instead of a beautifully packaged backtest.*