Synthetic data shows what you were solving for
what you'll learn · Why a strategy that matches its baseline exactly on synthetic data is informative — it tells you what problem your data lacks, not just what the strategy doesn't do.
Shipped a chop-filter strategy designed to gate cross-sectional momentum during noise periods. Ran it on the synthetic harness. Got numbers identical to baseline — the threshold never gated because the synthetic data has no chop. That's not a failure of the strategy. It's a feature of synthetic data: when a wrapper has no effect, you've learned which problem the wrapper solves and which problem your data has.
XsMomentumWithSpreadFilterStrategy ships a chop filter on top of
cross-sectional momentum: only trade when the long-bucket mean score
minus the short-bucket mean score exceeds a threshold. Designed
to solve the problem the
strategy-shape-beats-factor-count
note named — chop periods where cross-sectional rankings have no
real signal.
Ran it against the 12-arm harness. Result:
baseline: mean +1.001, stdev 0.581, min +0.173, max +1.656
spread_filter: mean +1.001, stdev 0.581, min +0.173, max +1.656Identical to baseline. Every digit. The chop filter never gated, because the synthetic data never had chop. The strategy collapsed to bare cross-sectional momentum.
Why this is informative, not embarrassing
A first reading would say: the strategy didn’t work; ship it again with different parameters. A more useful reading: the strategy DID work — the gate was correctly inactive — and the synthetic data revealed it has no cross-sectional chop.
The --fomc-drift-bps 50 synthetic was designed to test
event-aware wrappers. It has strong, directional cross-sectional
dispersion by construction — each FOMC day adds a 50 bp drift
component signed by recent momentum, which guarantees a wide
top-to-bottom spread in the panel rankings. The data doesn’t
HAVE the regime the chop filter solves.
That’s not a bug in the strategy. It’s a feature of the synthetic: every wrapper’s behaviour on it tells you what the wrapper protects against in the opposite regime. The spread_filter strategy on the no-chop synthetic is the absence of an event-window-style change in the orders — proof the gate is dormant when the regime doesn’t call for it.
The generalisable rule
When a strategy matches its baseline arm exactly on synthetic data, ask three questions:
-
What regime is the gate designed to detect? For spread_filter: cross-sectional chop (small long-short spread). For TSMom’s threshold: time-series chop (small per-symbol absolute return). For event blackout: pre-event windows. For the citation contract gate (ADR-0058): cold inner.
-
Does the synthetic data ever enter that regime? For
--fomc-drift-bps 50: no — cross-sectional spread is always large. For isotropic shocks: no — TSMom threshold rarely breached. For the no-FOMC half of the calendar: yes — event blackout never engages. -
If the answer to (2) is no, the gate is correctly dormant. The strategy’s identical-to-baseline behaviour is the gate working as designed. Ship it. The validation comes on data where the regime actually occurs.
What this rules out
This argument applies to wrappers — strategies that decorate a baseline with a conditional gate. It doesn’t apply to score strategies (two-factor, three-factor, four-factor) where a matched baseline number means the factor weighting genuinely didn’t add information.
Concretely:
| Strategy type | Identical-to-baseline means |
|---|---|
| Wrapper / gate (spread_filter, blackout, drift, damping) | Gate dormant — the regime isn’t present. Informative non-result. |
| Score combination (two_factor, three_factor, four_factor) | Factors didn’t matter on this data. Less informative. |
| Different shape (ts_momentum) | Different bucket — direct comparison less meaningful. |
The catalog now has both shapes; the harness measures both contributions; the discipline rule is: read identical-to-baseline results through the strategy’s design intent, not as a failure.
What it doesn’t mean
- Not “synthetic results are useless.” They’re more useful when they show no change, because the absence is the signal.
- Not “ship every strategy regardless of synthetic results.” A wrapper that matches baseline on synthetic AND matches baseline on real data is — at best — operationally inert. The synthetic result is the first test; real data is the load-bearing one.
- Not “the chop filter is well-tuned.” spread_threshold=0.04 is too low for this synthetic. The right next move is to bump it (to 0.10 or higher) and re-run; the threshold needs to match the regime the wrapper claims to detect.
The platform’s discipline rule:
When a wrapper strategy matches its baseline on synthetic data, verify the gate is dormant (not silently passing through). The non-result is a passed correctness test, not a failed performance test. The performance test runs against data that actually triggers the regime.