Article 2 of 7 — Building Momentum Systems Without Fooling Yourself. Previous: Architecture: Primitives, Layers, and the Judge · Next: Labelling: Triple Barriers in Volatility Units

Article 1 described a library of primitives and a judge that cannot express a strategy. That architecture makes trials countable. It does not count them. Nothing in the software stops me from running forty variants of a breakout on a quiet afternoon, keeping the one I liked, and remembering the episode as “I tested a breakout”. The gap between the trials you ran and the trials you admit to is where the “Illusion” posts said most strategies die. This article is about the bookkeeping that closes it: a register of hypotheses written before the data is touched, and a funnel that decides which hypotheses may touch it at all.

Counting before, not after

The usual practice, when anyone bothers, is retrospective: at the end of a project you reconstruct what you tried and produce a number for the multiple-testing correction. The number is always wrong, and always in the same direction, because memory drops failures and never invents extra ones. A trial count assembled after the fact is a lower bound produced by the person with the strongest incentive to keep it low.

Counting before means the register entry exists the moment an idea becomes concrete enough to test, and the entry is what authorizes the test. If the entry precedes the run, a disappointing run cannot be quietly forgotten, because the entry is already there. The register is append-only: entries can be superseded, withdrawn, or closed with a verdict, never edited in place or deleted. An embarrassing hypothesis is a tested hypothesis, and the count goes up either way.

Practically, the register is a database, and what I read from it are generated cards: read-only views, one per hypothesis, showing what was claimed, what was fixed in advance, which stage it reached, and what verdict it received. The cards are generated rather than hand-written for the obvious reason. If the card were the source of truth, I could edit it. As a projection of an append-only log, I cannot without leaving a trace I would have to explain to myself later.

The funnel

Not every idea deserves to become a trial; the funnel exists to say no early and cheaply. Ideas move through gates G0 to G8, and the first two — admission — are where most of them stop.

G0 is the mechanism question. Why should this work? What economic or behavioural reason would make a market exhibit the effect the signal is supposed to capture, and does that reason survive being written down in one paragraph without reference to a chart? “It looked good on crude last year” is not an answer. “Trend-following captures slow diffusion of information into prices, and should be weaker where information flows faster” is an answer, and one that also says where the hypothesis ought to fail.

G1 is the two-stage admission filter. First stage: a mechanical checklist. Is the mechanism stated? Is the universe named — which of the twenty-five markets, and why those? Is the regime scope named, or is the claim unconditional? Is the metric fixed, and is it the primary metric rather than whatever looked best? Is the labelling scheme the registered one, or does this hypothesis ask for a new one, in which case it is two hypotheses? Has a comparable entry already been closed, and what makes this one different? The checklist has no opinion; it is a form, and its fields are filled or they are not.

Second stage: human judgment, which for a solo operator means me, on a different day if I can manage it. A hypothesis can satisfy the checklist while being an obvious re-skin of something that already failed. The judgment question is blunter: if this passes, will I believe it, and if it fails, will I accept it? An idea whose failure I would not accept is a wish, not a hypothesis, and wishes do not go into the register.

Gates G2 through G8 follow the order of the series: labelling confirmed, unfiltered baseline established, regime features evaluated as features, secondary model validated under purged cross-validation, random benchmark run, return over drawdown reported as a distribution, walk-forward confirming or not. Each gate records whether the hypothesis is still alive. One that dies at G4 is a closed trial, not a draft.

Enumerating the space before searching it

A register counts what was tried. A catalog describes what could be tried, and the difference matters for the correction below.

The catalog is organized as mechanism × universe × regime. Mechanisms are the handful of economic stories a momentum system can plausibly rest on; universe is the markets or market groups a mechanism might apply to; regime is the market state under which it might apply — trending versus choppy, high versus low volatility. Every hypothesis lives in a cell of this grid, and the grid is written down before any cell is tested.

Two things follow. I can see how large the search space is, which stops me pretending the three cells I tested were the only three available. And an idea that fits no cell is a signal: either the catalog is missing a mechanism, or the idea has none, and both are worth knowing early.

What counts as one trial

Here the framework takes a position that is defensible but not free of judgment.

A hypothesis that touches data and is evaluated on the metric is a statistical trial. Full stop. That includes the ones that die early and the ones I would rather forget.

A descriptive batch is different. Running the six baseline probes of Article 4 identically across all twenty-five markets, with a fixed exit structure, no filters, and no selection, produces a hundred and fifty numbers. It is tempting to call that a hundred and fifty trials. I do not, because no selection is performed on the output. The batch is a map: nothing is kept or discarded on the basis of what it shows; it exists so that later, selective hypotheses have a baseline to be compared against. The register records it as one family-level trial: the family of textbook trend entries, evaluated descriptively.

The defence is that multiple-testing corrections exist to account for selection, and selection is precisely what the batch does not do. If I picked the best three of the hundred and fifty and registered those, the batch would have become a selection step and the count would have to reflect it. The rule is not “batches are free”. It is “a batch is one trial as long as nothing downstream is chosen because of it”, and when that condition breaks, the accounting changes.

Where is the judgment call? Exactly there. I decide whether a later hypothesis was “inspired by” the batch or “selected from” it, and the line is not sharp. The long-short asymmetry on US equity index futures came out of the baseline batch. When I register a hypothesis that trades only the long side there, is that one new trial, or should the numbers I looked at be charged to it? My answer: it is a new trial, and it must state its mechanism — equity drift — independently of the batch, so that it would have been admissible without ever having seen the map. That is a rule I apply to myself. It is not one a stranger could verify.

Feeding the correction

The reason any of this matters is Part 2 of the “Illusion” posts. The deflated Sharpe ratio of Bailey and López de Prado asks, in essence: given that you ran N trials and reported the best, what is the probability that the best is better than nothing? In its general form,

DSR = Z[ (SR − SR₀) / σ̂(SR) ]

where SR is the reported Sharpe, σ̂(SR) is its standard error corrected for sample length and for the skewness and kurtosis of returns, and SR₀ is the expected maximum Sharpe from N independent trials with no true edge — a quantity that grows with N and with the dispersion of Sharpe ratios across trials. My primary metric is return over drawdown, not Sharpe, and Article 7 deals with that; but the logic — a benchmark that rises with the number of things you tried — applies to any metric, and N must come from somewhere.

It comes from the register. Statistical trials count one each; family-level batches count one each; the dispersion of outcomes is read from the closed entries. Nothing is reconstructed from memory. If the register says N is fourteen, a good-looking fifteenth hypothesis is judged against the expected best-of-fifteen under the null, not against zero. Usually that is enough to turn “promising” back into “noise”, which is the correction doing its job.

What the register does not see

A register trusted too much is worse than none, so the limits deserve stating.

It undercounts. Every time I open a chart, notice a signal looks wrong on one market, and decide not to bother registering a hypothesis about it, I have spent a degree of freedom the register never sees. The funnel reduces these glances by making it cheap to register and awkward to skip, but cannot eliminate them. The count I feed to the correction is a floor, and I try to remember that when the corrected number looks good.

And I am my own auditor. The second admission stage, the “inspired by” versus “selected from” decision, the verdict at each gate — all of it is one person checking one person’s work. I cannot separate proposer from reviewer, so I lean on the parts of the system that do not need me to be honest: the append-only log, the generated cards, the timestamps, the checklist whose fields are filled or not. I built those constraints because I do not trust the person operating them at four in the afternoon after a disappointing run.

The register tells me how many hypotheses I ran. The next article is about what each of them is asked to predict. Fixed-horizon returns are the wrong target for a momentum signal; the triple-barrier method is the right one, and defining its barriers in volatility units lets one specification cover twenty-five markets without spending a degree of freedom on each.

References

Bailey, D. H. and López de Prado, M. (2014). “The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and Non-Normality.” Journal of Portfolio Management.

López de Prado, M. (2018). Advances in Financial Machine Learning. Wiley.

quanthedgeai — the research arm of algosworksai. Systematic research on mid-frequency instruments.