The single-seed lead was a fluke
what you'll learn · Why a single-seed backtest leaderboard is a sample of one, why the leader you read off it is more often a high-stdev arm having a good day than the actual best arm, and what reading the multi-seed sweep changes.
PR-shipping the single-seed FOMC compare result said long_only was the leader at Sharpe 1.88. Re-running 5 seeds at the same config: long_only's mean Sharpe is 0.81 — below the 1.00 baseline — with stdev 1.49 and a range of −0.93 to 2.45. ts_momentum is the actual consistent leader at mean 1.27 across the sweep. The note's previous reading was the result you get when the noise lines up.
[[event-gates-cost-when-the-event-has-edge]] read the single-seed
FOMC compare at /strategies/compare/ (seed 7, fomc_drift_bps=50)
top to bottom: long-only led at Sharpe 1.88, baseline trailed at
1.24, every event gate underperformed. The note interpreted the
gap as “strategies that maintain exposure beat strategies that
give it up on data with positive event-day drift.”
That reading was right about the shape — event gates do underperform when the event has directional edge. It was wrong about the leader.
What the 5-seed sweep shows
Re-running every arm across 5 consecutive seeds at the same pinned config, sorted by mean Sharpe:
| Arm | Mean | Stdev | Min | Max |
|---|---|---|---|---|
| ts_momentum | 1.27 | 0.78 | 0.63 | 2.14 |
| vol_weighted | 1.05 | 0.74 | 0.26 | 2.15 |
| blackout | 1.03 | 0.48 | 0.36 | 1.61 |
| spread_filter_tuned | 1.02 | 1.00 | −0.26 | 2.10 |
| damping | 1.01 | 0.49 | 0.34 | 1.62 |
| baseline | 1.00 | 0.58 | 0.17 | 1.66 |
| spread_filter | 1.00 | 0.58 | 0.17 | 1.66 |
| long_only | 0.81 | 1.49 | −0.93 | 2.45 |
| equal_risk_long_only | 0.81 | 1.49 | −0.93 | 2.45 |
| ts_momentum_aux (drift / reversal / factor variants) | < 0 | … | … | … |
The leader on seed 7 (long_only at 1.88) sits 7th on the multi-seed mean. Its 1.49 stdev is more than 2.5× the baseline’s noise floor. On seed 9 it was at −0.93. The seed-7 number wasn’t a result — it was the seed-7 outcome of a high-variance arm.
ts_momentum (time-series momentum, absolute-threshold) is the
arm that consistently outperformed. Mean 1.27 vs baseline 1.00 is
a real-looking +0.27 Sharpe. Its stdev (0.78) is wider than
baseline’s, but it never went deeply negative — the min across
5 seeds was +0.63.
Why long-only’s variance is so much wider
[[long-only-buys-asymmetric-exposure]] foreshadowed this exact shape on the prior, narrower FOMC compare:
Mean Sharpe was −0.196 vs the long-short baseline. The interesting number wasn’t the mean — it was the stdev: 1.487 across 5 seeds, more than double baseline’s 0.581.
That note named the mechanism: removing the short leg removes the dollar-neutral diversification that flattens per-seed dispersion. The exposure asymmetry that lets long-only catch the upside on a positive-drift seed also amplifies the downside on a negative-drift one. The note’s prediction of “more than 2× baseline stdev” lands exactly here (1.49 vs 0.58).
What the prior note couldn’t say — because it ran a narrower
configuration — was that on fomc_drift_bps=50, the +0.64 lead
on the best seed and the −2.13 deficit on the worst seed are the
same arm under different luck. A single-seed leaderboard sees
only the best half of that distribution if it happens to pick a
good seed.
What the gates do at the multi-seed reading
The event gates (blackout, damping) cost ~0.11–0.14 Sharpe vs
baseline on seed 7 in the single-seed reading. Across the 5-seed
sweep, they’re statistically indistinguishable from baseline:
mean 1.03 / 1.01 vs baseline 1.00, with stdevs (0.48, 0.49) that
are lower than baseline’s (0.58). The gates aren’t hurting on
average — they’re flattening dispersion, which is the protection
they were designed for, even on data where the events also have
directional edge.
That’s a substantively different finding from the single-seed one. The blackout/damping arms look like “small Sharpe cost for real variance reduction” on the sweep — that’s a positive result for a risk-reduction gate, not the +/-0.11 the single seed was implying.
The methodology correction
[[the-baseline-arm-you-forgot]] makes the case for the third arm: run the gate’s inverse alongside the gate to read whether the event window has directional edge. The multi-seed sweep is the temporal counterpart: run the same N arms across many seeds to read whether any arm’s lead is signal or sample-of-one luck. Both are correctives the two-arm-single-seed reading structurally can’t supply.
[[synthetic-data-shows-what-you-were-solving-for]] still applies on top of both. The multi-seed sweep doesn’t make the synthetic real. What it does is separate “what’s robust under the synthetic data generator” from “what was lucky on a particular draw of it.” A live-data leaderboard would face the same fluke risk; the multi-seed sweep is one of the few defences available before live capital arrives.
What lands on the live book
The promotion path the platform uses (ADR-0019) requires a named
human approver between propose and live. The right rubric for
that human is: did the arm beat baseline on the mean across the
multi-seed sweep, and does its min not go deeply negative? On
this configuration, only ts_momentum clears both — and its
seed-7 single-seed number (0.63 — well below baseline) is what
the human would have seen if they’d looked at one number.
The lesson the previous note missed: don’t promote off a single-seed leaderboard, even with the visual-axis check, even with the gate / counterfactual triplet. Add the seed sweep.
The visualised version of every number above lives at
/strategies/compare/ — under the new
“Across 5 seeds · mean ± stdev” section.