How to know if your trading backtest is overfit

Overfitting is when a strategy is tuned so tightly to historical data that it has memorised the past instead of capturing a real edge. The tell is a backtest that is too clean — smooth curve, tiny drawdown, dozens of parameters.

Red flags

• Many parameters, each finely tuned.
• Performance collapses if you nudge a setting slightly.
• One historical regime carries the whole result.
• A suspiciously straight equity line.

The honest tests

1. Out-of-sample. Hold back data the strategy never saw; if it only works in-sample, it's overfit.
2. Monte Carlo resampling. Resample the daily P&L into thousands of alternate sequences (a block bootstrap preserves short-term autocorrelation). Now you see a distribution of outcomes, not one lucky path.
3. Blow-rate metric. Across those simulated paths, what fraction end in a blow-out under real account rules? That annualized blow rate is the number that actually matters — far more than a headline return.
4. Parameter sensitivity. A robust strategy degrades gracefully as you vary settings; a fragile one falls off a cliff.

How Puravida Edge does it

Every strategy is validated on 12 months of empirical data, then resampled into 1,500 Monte Carlo paths over a 3-year horizon with a 5-day block bootstrap. We report percentile outcomes (P25/P50/P75) and an annualized blow rate, not just a best-case return — and we publish the methodology rather than a single hero curve. Full detail on the methodology page; outcomes per portfolio in the Pass Estimator.

For a quick second opinion on a set of results, the backtest quality checker scores them and flags the patterns that usually indicate overfitting.

What are some methods to avoid overfitting when backtesting a trading strategy?

Five methods, in order of how much they catch. Overfitting is fitting the specific history rather than the underlying behavior, and it is the default outcome of iterating on a single dataset — not an occasional accident.

1. Hold out data properly. Reserve a later period the strategy never sees during development, and test once. Testing repeatedly against the held-out set converts it into in-sample data, which is the most common way this method is defeated.

2. Check parameter sensitivity. Vary each parameter around its chosen value. A robust strategy shows a plateau — nearby values perform similarly. A fitted one shows a spike, where the chosen value works and neighbors collapse. The spike is the clearest single indicator of curve fitting.

3. Count your attempts. The number of variations tested is part of your statistics. Testing 500 combinations and reporting the best is guaranteed to produce an impressive result whether or not an edge exists. Fewer parameters and fewer iterations produce more trustworthy findings.

4. Reshuffle the sequence. Reorder your historical trades several thousand times and examine the distribution of outcomes. Your single historical path is one draw from that distribution, usually a favorable one. If most reshuffles produce unacceptable drawdowns, the backtest was flattered by ordering. The Monte Carlo simulator produces this distribution.

5. Demand a mechanism. If you cannot explain why the edge exists in terms of market behavior, it is more likely a coincidence in the data than a relationship. This filter is unfashionable and catches things the statistical tests miss.

FAQ

How can I tell if my backtest is overfit?

Watch for too-clean results, many finely-tuned parameters, and performance that collapses when you change a setting or test out-of-sample. A real edge degrades gracefully and survives unseen data.

What does Monte Carlo do for a trading strategy?

It resamples your daily P&L into thousands of alternate sequences so you see a distribution of outcomes — including the probability of a blow-out — instead of one historical path.

What's a block bootstrap and why use it?

It resamples in short blocks (e.g. 5 days) rather than single days, preserving short-term autocorrelation so the simulated sequences behave like real markets.

Which metric matters most for prop trading?

The annualized blow rate — the share of simulated paths that violate account rules — matters more than headline return, because surviving the drawdown rule is what gets you paid.