Part 2 of 2 — the remedies. Part 1 covered the intuition

Part 1 ended with a diagnosis: the gap between in-sample and out-of-sample Sharpe is the bill for the noise you memorized, and sequential optimization runs that bill up silently because its degrees of freedom are never recorded. This part is about paying the bill in advance — or, better, not incurring it.

There are three levers. Count the trials you actually ran. Correct the metric for that count. Restructure the research process so that the process itself, not just its output, gets validated. None of them is exotic; what is rare is applying all three with discipline.

1. Counting hidden degrees of freedom

The first honest step is admitting that “I tested three filters” is a fiction. The number that matters is the number of *forks* in the research path: every candidate condition, every threshold you nudged, every time you looked at an equity curve and decided what to try next. Gelman and Loken call this the garden of forking paths — the analysis you ran was one of many you *would have* run had the data looked different, and every one of those counterfactual branches inflates your result exactly as if you had run it.

You cannot reconstruct this number after the fact. So record it before. Concretely:

– Keep a research log: in which every backtest run is an entry, with the hypothesis, the parameter, and the metric. Not a summary — a log. If the tooling makes this automatic (a wrapper that writes a row per run), it will actually happen.
– Count thresholds as trials: A volatility filter tested at the 70th, 80th and 90th percentile is three trials, not one idea. Sweeping a threshold “to see what looks best” is a small grid search wearing a hypothesis as a disguise.
– Estimate the effective number of trials: Correlated trials do not count as independent ones. Ten filters that all proxy the same regime variable carry perhaps two or three independent bets. López de Prado’s approach is to cluster trial returns by correlation and treat the number of clusters as the effective *N*. It is a rough estimate, but a rough estimate of the right quantity beats a precise count of the wrong one.

Even an honest log undercounts, because it cannot capture the filter you rejected after ten seconds of staring. Use it as a floor. If the floor already says fifteen trials, the true number is not fifteen.

2. Correcting the metric: the deflated Sharpe ratio

Once you have an estimate of N, the question becomes: given N attempts, how high a Sharpe ratio would I expect to find *by chance alone*? Any reported Sharpe has to be judged against that benchmark, not against zero.

Bailey and López de Prado formalize this as the deflated Sharpe ratio. The construction has two steps.

First, compute the expected maximum Sharpe under the null hypothesis of no skill, given N independent trials whose Sharpe ratios have variance V:

SR₀ ≈ √V · [ (1 − γ) · Z⁻¹(1 − 1/N) + γ · Z⁻¹(1 − 1/(N·e)) ]

where Z⁻¹ is the inverse standard normal CDF and γ ≈ 0.5772 is the Euler–Mascheroni constant. The exact form matters less than its behaviour: SR₀ grows with N, slowly, roughly with √(2 ln N), but it never stops growing. Twenty trials of pure noise on a strategy with annualized Sharpe volatility of 0.5 produce an expected best Sharpe of around 0.9. That is your baseline. A Sharpe of 1.1 after twenty attempts is not evidence of anything.

Second, ask how likely it is that your observed Sharpe exceeds SR₀ once you account for the sample length T and for the non-normality of the returns (skewness γ₃ and kurtosis γ₄, which fatten the distribution of the Sharpe estimator):

DSR = Z[ (SR − SR₀) · √(T − 1) / √(1 − γ₃·SR + (γ₄ − 1)/4 · SR²) ]

DSR is a probability. Below 0.95, you do not have a strategy; you have a lottery ticket that paid out in-sample. This is a far harsher test than the usual “Sharpe above 1 and the curve looks smooth,” and it is harsh in exactly the right way: it penalizes the number of attempts, the shortness of the sample, and the fat tails, which are the three things sequential optimization exploits.

Two caveats. DSR needs an N and a V, and both come from the log you kept in step 1; feed it a flattering N and it will flatter you back. And it corrects for multiple testing, not for structural non-stationarity, a filter that genuinely worked from 2009 to 2019 and then stopped will pass DSR and still fail you. Different problem, different tools.

3. Restructuring the process: hold-out and nested cross-validation

Corrections are for damage already done. The stronger remedy is to design the research so the damage cannot happen.

The hold-out, done properly

Everyone knows about hold-out sets. Almost nobody uses them properly, because the temptation to peek is enormous and the moment you peek the hold-out is gone. The rules that make a hold-out mean something:

– Lock it before the first backtest. Not after the baseline, not after the first filter. Before anything.
– Touch it once The hold-out is a single-shot test of the final strategy. If the result disappoints and you go back to modify the strategy, the hold-out is now in-sample and you need a new one you do not have.
– Pre-register the decision rule. Write down, before looking, what constitutes a pass: “out-of-sample Sharpe ≥ 60% of in-sample Sharpe” or “DSR on the hold-out ≥ 0.9.” Without a pre-specified rule, you will find a way to call any result acceptable.
– Size it for the question. A six-month hold-out on a strategy that trades twice a week has maybe fifty trades; it cannot distinguish a Sharpe of 0.5 from one of 1.5. If the hold-out is too short to detect the degradation you fear, it is decoration.

Nested cross-validation

The hold-out validates a strategy. It does not validate the procedure that produced it, and the procedure is what you will reuse on the next instrument, the next idea, the next year. That is what nested cross-validation is for.

The structure has two loops. The inner loop does the research: on a training fold, it runs the whole sequential process, candidate filters, threshold selection, keep-or-discard decisions  and outputs a strategy. The outer loop evaluates that strategy on a fold the inner loop never saw, then repeats with a different partition. The result is a distribution of out-of-sample Sharpes *for the research process*, not for any single strategy.

This changes the question from “does this strategy work” to “does the way I build strategies produce things that work.” A process whose outer-loop Sharpe averages 0.2 while its inner-loop Sharpe averages 1.4 is a noise-memorization machine, regardless of how good any single output looks.

For financial time series the folds need two additional mechanics. Purging removes training observations whose labels overlap in time with the test fold, so that a trade opened in-sample and closed in the test period does not leak information across the boundary. Embargo adds a gap after each test fold so that serial correlation in features does not do the same. López de Prado’s combinatorial purged cross-validation (CPCV) implements both and additionally generates many test paths from the same data, which gives you a distribution of outcomes rather than a single backtest curve, a far more honest picture of what production might look like.

The cost is real: nested CV multiplies compute by the number of outer folds and forces you to automate the research process rather than drive it by hand. That second cost is the point. A process you can automate is a process whose degrees of freedom you can count.

The protocol, in one paragraph

Lock a hold-out before you start. Write down every candidate filter you intend to test, with the economic rationale for each, before running any of them; anything not on the list requires a new pre-registration, not a quick look. Log every run. Estimate the effective number of trials by clustering. Report the deflated Sharpe ratio, not the raw one, and treat DSR below 0.95 as a rejection. Run the whole procedure inside a nested, purged cross-validation and report the outer-loop distribution. Only then touch the hold-out, once, against a pre-specified pass criterion.

This is slower. It will kill most of the strategies that currently look good in your research notebook. That is not a side effect; it is the entire purpose. The strategies that survive it are the ones with a chance of surviving production — and the ones that don’t were never going to.

—

*Part 1: [The Illusion of the Filter That Works: Why Adding Conditions Sequentially Destroys Strategies](https://www.quanthedgeai.com/blog/the-illusion-of-the-filter-that-works-why-adding-conditions-sequentially-destroys-strategies/)*

**References**
– Bailey, D. H. and López de Prado, M. (2014). *The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and Non-Normality.* Journal of Portfolio Management.
– López de Prado, M. (2018). *Advances in Financial Machine Learning.* Wiley. (Chapters 7, 12, 14.)
– Harvey, C. R. and Liu, Y. (2015). *Backtesting.* Journal of Portfolio Management.
– Gelman, A. and Loken, E. (2013). *The Garden of Forking Paths.*