Purged walk-forward vs ordinary cross-validation: preventing leakage in financial time-series ML

The most expensive mistake in financial machine learning is not a bad model. It is a good-looking backtest that leaked. The model was never as good as the numbers said, but you find that out with real money instead of in validation. This note is about the validation discipline that stops that from happening, drawn from the validation layer of my quantitative trading research platform. It is entirely about methodology. None of it makes a strategy profitable; it makes your evaluation honest, which is a different and more modest claim.

Why ordinary k-fold cross-validation leaks

Standard k-fold cross-validation assumes your samples are independent and identically distributed. Shuffle the rows, split into folds, train on some, test on the rest, and each test fold gives an unbiased estimate of out-of-sample performance. On financial time series that assumption is wrong in two ways at once, and both inflate your results.

The first is serial correlation. Adjacent observations in a price series are highly dependent. If you shuffle and split, a test-fold row sits temporally between two training rows it strongly resembles. The model does not need to generalize; it can nearly interpolate. Test performance looks like memorized neighbors, not genuine prediction.

The second, and worse, is label-horizon overlap. In practice you rarely label a bar by its next-tick return. You label it by what happens over a forward horizon — the return over the next N bars, or whether a price barrier is touched within a window. That means the label attached to time t is computed from data spanning t to t + h. The label at t and the label at t + 1 are built from almost the same future window. They are not independent observations; they are overlapping views of one stretch of the future.

Now the leak is concrete. Put t in training and t + 1 in test, and the training label already contains most of the information in the test label, because their forward windows overlap. The model has effectively seen the test answer through the back door. Nothing in ordinary k-fold prevents this, and the resulting backtest can look excellent while measuring almost nothing.

Triple-barrier labeling makes the overlap explicit

The overlap is easiest to see with triple-barrier labeling, a scheme from Marcos López de Prado’s Advances in Financial Machine Learning. For each event you set three barriers: an upper profit-taking barrier, a lower stop-loss barrier, and a vertical time barrier. The label is whichever barrier the price touches first. It is a realistic way to label because it mirrors how a position actually exits.

It also means every label spans an interval — from the event time to whenever a barrier is hit — rather than sitting at a single instant. When events are sampled densely, those intervals overlap heavily. Two events a few bars apart can resolve against overlapping, sometimes identical, future paths. The label spans are the leakage surface, and you can no longer pretend each labeled example is one independent draw.

The fixes from the field

The techniques below are standard in the quantitative-finance literature; the credit for purging, embargo, and CPCV goes to López de Prado. I did not invent any of them. The engineering contribution is implementing them correctly and wiring them into a promotion gate, not the methods themselves.

Purging

Purging removes from the training set any observation whose label span overlaps in time with the label span of any test observation. Concretely, if a test event’s outcome is determined over [t, t + h], you drop every training event whose own label interval intersects that window. This severs the specific overlap that leaks the answer. The cost is fewer training samples, especially near the test boundary, which is a price worth paying for an estimate that means what it says.

Embargo

Purging handles direct overlap, but serial correlation leaks a little further than the label horizon. Embargo adds a small buffer after each test interval and drops training observations that fall inside it, even if their label spans do not strictly overlap. The embargo width is a modeling choice, typically a small fraction of the total span; it exists to absorb residual autocorrelation that purging alone leaves on the table.

Purged walk-forward

Combine purging and embargo with a forward-only split and you get purged walk-forward validation: always train on the past, test on the future, and purge and embargo the boundary between them. This respects the arrow of time — no test row is ever earlier than a training row — while cleaning the overlap that plain walk-forward would still leak at the seam. It is the honest default for a single evaluation path.

Combinatorial purged cross-validation

Walk-forward gives you exactly one backtest path, and one path is a single sample of a noisy process. Reorder history slightly and the number changes. Combinatorial purged cross-validation (CPCV) addresses this by taking N time blocks and testing on every combination of k of them, purging and embargoing around each test group. Instead of one out-of-sample path you get many — each a different partition of history into train and test — and therefore a distribution of performance rather than a point estimate. That distribution is what tells you whether an edge is stable or an artifact of one lucky ordering.

Sample uniqueness and concurrency weighting

There is a subtler correction that matters even after purging. When labels overlap, dense stretches of the series contribute many near-duplicate examples, and the model over-weights those periods simply because they are crowded. The fix is to weight each sample by its uniqueness — inversely to how many other labels are concurrent with it over its span. Overlapping labels stop counting as independent evidence, both in training and in any metric that averages over samples. Without this, effective sample size is much smaller than the row count suggests, and every downstream statistic is overconfident.

Honest evaluation after the split

Leakage-resistant splitting is necessary but not sufficient, because there is a second inflation source: selection under multiple testing. If you try a hundred configurations and report the best Sharpe ratio, that maximum is biased upward regardless of how clean each individual backtest was. The more strategies you try, the higher the best-of looks purely by chance.

The countermeasures come from the same literature. The probabilistic Sharpe ratio (PSR) asks how confident you can be that the true Sharpe exceeds a benchmark, given track-record length, skew, and kurtosis. The deflated Sharpe ratio (DSR) goes further and discounts the observed Sharpe for the number of trials you ran and the variance across them — directly penalizing the multiple-testing search that produced your best candidate. Reporting DSR alongside the raw number, and pairing it with bootstrap confidence intervals and cost-sensitivity sweeps, is what keeps a promising backtest from being an accident of selection.

What this does and does not buy you

I want to be exact about the claim, because it is easy to overstate.

Purging, embargo, purged walk-forward, CPCV, uniqueness weighting, and deflated Sharpe together remove the main ways a backtest lies to its author. They give you an out-of-sample estimate that is not silently contaminated by the future, and a performance figure that has been discounted for how hard you searched. That is the entire value on offer: an evaluation you can trust to be measuring what it claims to measure.

None of it makes a strategy profitable. Markets are non-stationary; a relationship that held out of sample in your cleaned validation can decay or invert the moment it meets live capital. A rigorous validation method is a necessary condition for taking a result seriously, never a sufficient one for expecting returns.

Limitations

The discipline has real costs and real gaps. Purging and embargo shrink the training set, sometimes sharply, so on short histories you trade leakage for variance. Embargo width and CPCV block count are themselves choices that can be tuned until the answer flatters you — the meta-overfitting risk moves up a level rather than disappearing. Deflated Sharpe depends on an honest count of trials, and honest counting is a discipline, not an equation; undercount your experiments and DSR is optimistic again. And all of it validates against the past. The one thing no split can protect you from is a regime that has not happened yet. This is methodology for making evaluation honest. It is not financial advice, and it is not a claim of profitability.