The Overfitting Problem
Give a computer enough parameters to optimize and it will find a combination that generates spectacular returns on historical data. This tells you almost nothing about the future. The strategy has memorized the past, not learned from it. This is called overfitting, and it is the single most common reason backtested strategies fail in live trading.
Bailey, Borwein, and Lopez de Prado published a paper in the Notices of the American Mathematical Society titled "Pseudo-Mathematics and Financial Charlatanism." Their key finding: high simulated performance is easily achievable after testing a relatively small number of alternative strategy configurations. Under memory effects, backtest overfitting leads to negative expected returns out-of-sample. Not zero. Negative. Meaning the overfit strategy is expected to lose money going forward.
The same team published "The Probability of Backtest Overfitting" in the Journal of Computational Finance in 2015, introducing a formal framework for calculating how likely it is that any given backtest is overfit. Their Deflated Sharpe Ratio corrects for selection bias under multiple testing and non-normal returns.
Out-of-Sample Testing Is Non-Negotiable
The minimum standard for any backtest is out-of-sample validation. Split your data into at least two periods. Develop your strategy on the first period (in-sample) and then test it on the second (out-of-sample) without any modifications. If performance degrades significantly on out-of-sample data, the strategy is overfit.
Walk-forward analysis takes this further. Instead of a single split, you repeatedly train on a rolling window of data and test on the next period. Train on years 1-3, test on year 4. Train on years 2-4, test on year 5. And so on. This simulates how the strategy would actually be used in practice, with parameters periodically re-optimized on recent data.
The ratio of out-of-sample to in-sample performance is the most important metric. A strategy generating a 2.0 Sharpe ratio in-sample and 1.4 out-of-sample has a 30% degradation, which is actually quite reasonable and suggests a genuine edge. A strategy generating 3.0 in-sample and 0.5 out-of-sample is almost entirely overfit and should not be traded. Reports indicate over 90% of academic strategies fail in live trading, and backtest Sharpe ratios of 3.0+ commonly become negative in production.
Transaction Costs and Slippage
A strategy that trades 50 times per day needs to clear a much higher bar than one that trades twice per month. Every trade incurs costs: the bid-ask spread, exchange fees, and slippage.
Kaiko Research data from 2024 shows Binance BTC-USDT average spread at around 0.0014 basis points, while Coinbase BTC-USD averages around 0.086 basis points. On smaller altcoins, spreads can reach full percentage points. Among institutional brokers, negative TWAP slippage of -1 to -2 basis points is typical in crypto.
Exchange fees add up: Binance charges 0.10% maker/taker on spot; OKX 0.08% maker, 0.10% taker; Coinbase Advanced ranges from 0.00-0.40% maker and 0.05-0.60% taker depending on volume tier. A realistic backtest must account for all of these costs, and many amateur backtests do not.
Survivorship Bias
If your universe only includes currently listed assets, your backtest is biased. Companies that went bankrupt, tokens that went to zero, and instruments that were delisted are missing from the data. Since these tend to be the worst performers, their absence makes every long strategy look better than it actually was. In crypto, the effect is severe because hundreds of tokens launched in 2017-2018 went to zero and no longer appear in most data feeds.
A proper backtest uses point-in-time data that includes all securities available at each historical moment, including those that subsequently disappeared.
Drawdown Duration Matters
Maximum drawdown tells you the worst peak-to-trough decline. But drawdown duration is equally important for practical trading. A strategy might have a maximum drawdown of only 15%, but if that drawdown lasted 18 months, most traders would abandon it long before it recovered. Knowing both the depth and duration of historical drawdowns helps you assess whether you can actually stick with a strategy through its inevitable losing periods.