Article 7 of 7  Building Momentum Systems Without Fooling Yourself. Previous: Meta-Labelling and Bet Sizing

Six articles in, a candidate strategy exists: a registered label specification, a baseline probe, a regime gate evaluated as a feature, a secondary model that sizes bets, and a row in the register timestamped before any of its results. This last article answers the question everything else was built to make answerable: what number do I believe, and how do I decide whether the strategy is allowed to trade?

The short answer is that there is no number. There is a distribution, a set of diagnostics, and a pre-registered rule.

Why return over drawdown

The platform reports a Sharpe ratio, and I do not use it. Part of the reason is specific: the platform’s version divides by an “account size required” figure derived from the backtest’s own worst excursion, so the denominator is a function of the realized path rather than of return variance, and the result is neither Sharpe nor anything else with a known distribution. But even a correctly computed Sharpe would be the wrong primary metric for my situation.

Sharpe assumes variance is the thing you pay for. That holds for an allocator with many uncorrelated strategies and capital indifferent to the ordering of returns. It does not hold for a solo operator running a small book. I am not ruined by variance. I am ruined by a drawdown deep enough, or long enough, that I stop trading — because the account forces it or because I lose the nerve to follow the system through the trough. Two strategies with identical Sharpe and different drawdown profiles are not equivalent to me; one of them I can actually hold.

So the primary metric, registered before the first run, is return over maximum drawdown, on an equity path reconstructed by the Python judge from raw prices and contract point values. It is a cruder statistic than Sharpe, with worse sampling properties. That is why the next section exists.

The number is a distribution

A single realized backtest produces a single return/drawdown ratio, and that ratio is a statement about one path. Maximum drawdown is an extreme-value statistic: it depends on the order in which the losing trades happened to cluster. Reorder the same trades and it changes by a factor that would embarrass anyone who had put the original figure in a report.

The common fix is to shuffle the trade list and rebuild many equity curves. I do not do this, because shuffling destroys the structure that produced the drawdown. Losses cluster because volatility clusters; a momentum strategy’s bad periods coincide with regime transitions, and those are not independent draws. Shuffling yields a comfortable, symmetric distribution of outcomes the market has never once delivered.

Instead the judge fits a path-generating process to the reconstructed strategy returns — heavy-tailed innovations under conditional volatility, in practice a Student-t distribution driving an EGARCH variance — and simulates many paths of the same length. Each path gets its own return/drawdown ratio. What I report, and what the pass rule is written against, is a quantile of that distribution, not the realized value. The realized backtest is one draw from that process, and it usually sits above the median of the simulated paths, which is what a single number hides: it is the survivor, the one you chose to look at.

The random benchmark

Kevin Davey’s argument is simple and I have not found a way around it: a strategy is only interesting relative to what random entries would have done with the same exit structure, the same holding characteristics, and the same markets. A trend strategy on an asset with drift shows a profit whether or not the entry contains information. Article 4 found exactly this on US equity index futures, where the long side of every probe works and the short side does not.

The detail that matters is that the random-entry strategies are run across all twenty-five markets simultaneously, so each random portfolio path carries the real cross-market correlation structure of the period. Running the benchmark market by market and combining the results would understate the variance of the portfolio outcome, because equity indices move together, rates move together, and in a crisis everything moves together. The benchmark has to be as correlated as the strategy is.

This gives a chance distribution not only for the portfolio’s return/drawdown but for breadth: the number of markets on which a random-entry strategy comes out ahead. Breadth is where judgment usually sneaks in — “it works on most of the universe, that is surely robust” — and the benchmark answers the question that sentence forgets to ask: on how many markets a coin would have won under the same conditions. In a period of strong drift the answer is uncomfortably high.

The benchmark is a diagnostic, not a gate. A strategy that clears the chance distribution comfortably has told me its entries carry information; one that does not has told me the exit structure and the drift are doing the work. Neither is a verdict on its own. It informs the human judgment stage, and it goes into the register entry either way.

Overfitting: CPCV, PBO, and the deflated ratio

The secondary model of Article 6 was fitted under purged and embargoed cross-validation, which guards against leakage across overlapping labels. Validation asks a different question: not whether the model generalizes across folds, but whether the selection of this strategy over its alternatives would have.

Combinatorial purged cross-validation, from López de Prado, answers this by splitting the history into groups and evaluating every combination of training and testing groups rather than a single chronological split. The result is many out-of-sample performance paths, each from a different assignment of history to train and test. The probability of backtest overfitting, from Bailey, Borwein, López de Prado and Zhu, then asks: across those combinations, how often does the configuration that ranked best in-sample rank below the median out-of-sample? Near one half, selection was worthless; materially higher, selection was actively harmful — the sequential-optimization signature of the “Illusion” posts, observed from the inside. The mechanics are in the paper and the book; the point is that the test is on the research process, not the strategy.

The deflated Sharpe ratio of Part 2 returns here with its input finally available. The register of Article 2 gives the count of trials that touched data, under the rule that a descriptive batch counts as one family-level trial. The Sharpe I deflate is the judge’s own, computed on the reconstructed path, and the deflation uses the number of trials implied by the register together with the variance of Sharpe across them. The output is a probability that the observed Sharpe exceeds what the best of that many trials would show under no edge. It is reported alongside return/drawdown, not instead of it: one answers a question about selection, the other about survivability.

Walk-forward as confirmation, and what “pass” means

Pardo’s walk-forward analysis — fit on a window, trade the next, roll forward — is the final step, deliberately. Used earlier, walk-forward becomes a development tool, and a development tool is an optimizer. If I ran it on ten candidate configurations and kept the one with the best out-of-sample efficiency, I would have spent ten degrees of freedom on the out-of-sample data and could no longer call it that. Walk-forward is run once, on the single configuration the funnel delivered, to confirm that behaviour under rolling refits is consistent with what the simulated distribution predicted. Disagreement is informative; agreement is not proof.

“Pass” is a rule written into the register entry at admission, before data. It names the quantile of the return/drawdown distribution that must clear a floor, the level the deflated Sharpe must reach, a ceiling on the probability of backtest overfitting, and a walk-forward consistency condition. I do not pick the thresholds after seeing how close the strategy came.

A strategy that fails does not go back to the workbench. It is closed in the register as a failed trial, diagnostics attached, and the count that feeds every future deflation goes up by one. This is the part that hurts, and the part that makes the framework work. The alternative — a small adjustment, a rerun, a pass on the second attempt — is the “Illusion” process wearing a lab coat. If the failure suggests a genuinely different mechanism, that is a new hypothesis, admitted through the funnel and counted like any other.

Closing the series

The “Illusion” posts described the bill for noise memorized: the gap between the in-sample number and what production delivers. This series has described the accounting system that keeps the bill from arriving. It is fair to end with what that system costs.

It costs speed. Most ideas die at the checklist, and the ones that survive take weeks rather than hours. It costs the pleasure of iteration, which is real. It costs infrastructure — a primitive library, a judge, a parity test, a register, a simulation engine — that produces no P&L by itself. And it costs a kind of honesty that is hard to maintain alone, because nobody will catch me if I quietly reopen a closed entry.

What it buys is a number I can act on: a distribution of return over drawdown, deflated for the trials I actually ran, benchmarked against chance in a universe with real correlations, confirmed once by a walk-forward I did not use to build it. Not a guarantee. A number whose failure modes I know.

The open questions remain open. I do not know that the horizon of the effects these probes target is stationary across twenty years; it may be shorter now than it was, and a label specification fixed a priori cannot chase it. Cross-market ranking of the probes is unstable enough that I am not sure “momentum” names one phenomenon across the universe rather than several. And meta-labelling contains overfitting at the level of the signal only to open a new door through the feature set, where sixty regime primitives are sixty ways to memorize. The register counts them. It does not make them fewer.

That is where the framework stands. It is slower than the alternative. It is supposed to be.

References. López de Prado, Advances in Financial Machine Learning (2018). Bailey & López de Prado, “The Deflated Sharpe Ratio” (2014). Bailey, Borwein, López de Prado & Zhu, “The Probability of Backtest Overfitting” (2017). Pardo, The Evaluation and Optimization of Trading Strategies (2008). Davey, Building Winning Algorithmic Trading Systems (2014).

quanthedgeai — the research arm of algosworksai. Systematic research on mid-frequency instruments.