Article 5 of 7 — Building Momentum Systems Without Fooling Yourself. Previous: Momentum Probes: The Baseline · Next: Meta-Labelling and Bet Sizing

This article is the direct answer to the two “Illusion of the Filter” posts. Those posts described a researcher who takes a baseline, adds a volatility filter, reruns, keeps it, adds a trend-alignment filter, reruns, keeps it, and ends up with an equity curve stitched together from lucky tails. The problem is not that filters are bad. Regime information is real; markets trend more readily in some states than in others. The problem is where the filter enters the process and who decides whether it stays. This article describes the arrangement I use instead: the same regime quantities, computed as primitives in the C# library, handed to a downstream model as features, evaluated once under a protocol fixed before the data was touched. The difference is not one of degree. It is a difference in kind, and I want to be precise about why.

The library, by family

There are around sixty regime primitives in the library. Sixty sounds like a lot until you notice they fall into four families, and within each family most members are variations on a single question asked at different scales or with different smoothing.

The first family is efficiency. The prototype is Kaufman’s efficiency ratio: how much net ground did price cover relative to the total distance it travelled to get there? A market that moves a long way and ends up where it started has low efficiency; one that walks in a straight line has high efficiency. Every member of this family asks whether the recent path looks like a trend or like a random walk that happened to drift.

The second family is persistence. These ask whether increments of the series are positively dependent, a move tends to be followed by more of the same or negatively dependent, or independent. The Hurst exponent is the classical one: an estimate of how the range of the series scales with the window, which grows at a known rate under a random walk and faster when the series is persistent. The variance ratio of Lo and MacKinlay is the other: the variance of multi-period returns compared to the variance of single-period returns scaled by the number of periods. Under a random walk the ratio is one; above one the series is trending, below one it is mean-reverting.

The third family is choppiness. Where efficiency asks whether the path is straight, choppiness asks whether the range is being consumed by directional movement or by oscillation. These are cousins of efficiency measures in intent but built from bar ranges rather than closes, and they respond differently to gappy markets.

The fourth family is volatility state. Not the level of volatility, which the labelling of Article 3 already normalizes away, but the state: is volatility high or low relative to its own recent history, rising or falling, in compression or expansion? Momentum behaves differently at the exit of a compression than in the middle of a long expansion.

I describe these by what they measure rather than by formula because the formulas are the least important thing about them. What matters is that each primitive computes one number per bar, the same way, for every one of the twenty-five markets, with no per-instrument tuning, and does nothing else.

The difference in kind

Here is the arrangement the “Illusion” posts described, stated as a procedure. Take an entry rule. Add a condition: enter only when a regime primitive is above some value. Run the backtest. Look. If it improved, keep the condition and try another; if not, change the value, or the primitive, or the direction of the inequality. Stop when the curve looks good.

Every step is a decision conditioned on the outcome, and every decision is a degree of freedom nobody wrote down. Part 1 called them hidden. Part 2 explained that once they are hidden you cannot correct for them, and the deflated Sharpe ratio has nothing to deflate by.

Here is the arrangement I use. The primitives are computed as features: at every bar, for every market, the library emits the full vector of regime measurements alongside the probe’s signal. Nothing is gated. The feature vector, the triple-barrier labels of Article 3, and the probe’s proposed side go to the secondary model of Article 6, which is trained and evaluated once under a cross-validation protocol specified before the run. The model decides how much weight each feature gets. I do not.

The degrees of freedom have not vanished. They have moved, and they have become countable. In the first arrangement they live in a sequence of keep/discard decisions, each made after looking. In the second they live in the composition of the feature set, which is declared up front, written into the register, and counted as a trial before the model sees a single label. The number of things that could have been tried is no longer the number that ran through my head while staring at a chart. It is the number of columns in a table, and the table has a timestamp.

That is the difference in kind. It is not that the second arrangement tests fewer hypotheses; a model that can weight sixty features explores a larger space than a researcher bolting on three filters. It is that the exploration happens inside a single, pre-registered evaluation, where the validation protocol sees all of it, rather than in a loop of separate backtests where the validation sees only the survivor.

Gate and direction

There is a second way regime primitives are used, and it looks superficially like a filter, so it needs careful distinction.

The “gate + direction” pattern splits a strategy into two questions. A regime gate decides whether the market is tradeable at all, whether the current state is one in which a trend-following probe has any business being active. The probe decides direction. The gate never looks at the probe’s signal, and the probe never looks at the gate.

What makes this a structural choice rather than a filter found in data is when it is decided. The gate’s role — that there is one, that it sits before the probe, that it is built from one specified family of primitives — is a design decision registered before any backtest. It comes from the mechanism catalog of Article 2: the hypothesis is “trend-following works in persistent regimes and not otherwise”, and the gate is the structural expression of that hypothesis. What the gate is not is a condition discovered by trying five primitives against an entry and keeping the one that raised the metric. If the gate hypothesis fails, it fails as one registered trial, and the answer is recorded, not iterated on.

The separation also means the two halves are judged against different benchmarks. The gate is judged on whether the periods it admits are better for trend-following than the periods it excludes, read against the random-entry benchmark of Article 7 rather than against zero. The probe is judged, as in Article 4, on direction. A gate that improves the equity curve only by reducing trade count is the arbitrary constraint Part 1 warned about, and the chance distribution exposes it immediately.

Sixty primitives are not sixty bets

A reader of Part 2 will have noticed a trap. If the feature set is a registered trial and the deflation depends on how many trials were run, sixty features must be a much bigger correction than three filters. That is true if the sixty are independent. They are not.

Efficiency and persistence measures are correlated because they detect the same thing from different angles. Choppiness measures are correlated with efficiency measures because both respond to the ratio of directional to total movement. Within a family, members at neighbouring scales are correlated almost by construction. The honest count is not sixty; it is the effective number of distinct things being measured.

So the feature set is clustered before it is used. Features that move together are grouped, the number of clusters is the effective number of features, and that number, not the raw count, enters the trial accounting. This is the same logic López de Prado applies to effectively independent trials in the deflated Sharpe ratio: what matters is the variance of outcomes across trials, and correlated trials contribute less of it. None of this makes the count small. It makes it honest, which is the only thing the framework promises.

The caveat I owe you

The feature arrangement contains the failure mode of the “Illusion” posts. It does not eliminate it. There is a residual channel: the choice of which features to include. If I ran the secondary model with feature set A, looked, then ran it with feature set B and kept the better one, I would have reintroduced sequential optimization one level up, not in the filter, but in the feature list. The model would be innocent; the researcher would not.

Two things guard against this. The feature set is itself registered as a trial under Article 2’s rules: a different feature set is a different hypothesis, it gets its own entry, and the entry counts whether the result is good or bad. And the secondary model is validated under purged, embargoed cross-validation, as Article 6 will describe, so that the evaluation of any one feature set is not contaminated by label overlap. Neither guard is perfect. A determined

Quant trader can still overfit through the feature set by running many registered trials and reporting the best. The register makes that visible and the deflation makes it expensive. That is as far as the method goes, and I would be lying if I claimed it went further.

Article 6 takes the feature vector and the labels and describes what the secondary model does with them: how it separates whether to act from how much, why that separation contains overfitting better than a single monolithic signal, and the specific way in which it does not.

References. López de Prado, M., Advances in Financial Machine Learning, Wiley, 2018. Bailey, D. H. and López de Prado, M., “The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and Non-Normality”, Journal of Portfolio Management, 2014. Lo, A. W. and MacKinlay, A. C., “Stock Market Prices Do Not Follow Random Walks: Evidence from a Simple Specification Test”, Review of Financial Studies, 1988. Kaufman, P. J., Trading Systems and Methods, Wiley.

quanthedgeai — the research arm of algosworksai. Systematic research on mid-frequency instruments.