Skip to main content
2026-05-22 · 4 min read · ← 6 · research · harness · leaderboard

One hundred seeds confirms and converges

what you'll learn · What an N=100 run adds over N=50: tighter means, narrower disagreements between arms, and clearer identification of arms that are statistically identical.

Re-ran the harness at N=100 to refine the 50-seed leaderboard. ts_momentum's lead over baseline holds (+0.132, 82% hit rate). The three three_clock variants converge to exactly +0.798 — what looked like distinct arms at N=10 are statistically identical at N=100. baseline and spread_filter are also indistinguishable. Two findings: ts_momentum is robust; many arms are duplicates.

PR #759 named follow-up #2: “The 100-seed run. N=50 already gives reasonable precision but N=100 would tighten the means by another 1/√2. Tractable in under 2 minutes on the current harness.”

The N=100 run took ~85 seconds and produced 2400 CSV rows. Here’s the top of the leaderboard:

arm                        mean    stdev    p25      p75     min      max    hit%
ts_momentum              +0.995   1.095   +0.448  +1.552  -2.358  +3.209   82%
baseline                 +0.863   1.077   +0.180  +1.588  -1.913  +3.368   79%
spread_filter            +0.861   1.076   +0.174  +1.588  -1.913  +3.368   79%
vol_weighted             +0.856   1.062   +0.210  +1.545  -1.847  +3.183   78%
blackout                 +0.851   1.067   +0.162  +1.624  -2.027  +3.151   81%
damping                  +0.838   1.068   +0.155  +1.625  -2.047  +3.135   81%
three_clock_vol_weighted +0.799   1.210   -0.034  +1.503  -2.337  +3.993   74%
three_clock_portfolio_vol+0.798   1.225   -0.102  +1.549  -2.461  +4.013   72%
three_clock_momentum     +0.798   1.232   -0.037  +1.606  -2.479  +4.078   74%
portfolio_vol_gate       +0.716   1.139   -0.117  +1.516  -1.562  +3.111   66%
three_clock_vol_regime   +0.680   1.237   -0.122  +1.393  -2.004  +3.706   72%
vol_regime_filter        +0.649   1.102   -0.181  +1.438  -1.928  +3.222   73%

Finding 1: ts_momentum is robust

N=10:  ts_momentum +1.173 vs baseline +0.999 (Δ +0.174)
N=50:  ts_momentum +0.910 vs baseline +0.767 (Δ +0.143)
N=100: ts_momentum +0.995 vs baseline +0.863 (Δ +0.132)

The Δ shrinks slightly from N=10 to N=100 but stays robust at +0.13 Sharpe. Hit rate jumps from 76% (N=50) to 82% (N=100) — ts_momentum positively-rewards on more seeds as the seed count grows.

The mean’s stdev should fall as 1/√N. Empirically:

  • N=10 stdev: ~1.0 → 1-stdev confidence interval ~0.32
  • N=50 stdev: ~1.1 → CI ~0.16
  • N=100 stdev: 1.095 → CI ~0.11

The Δ +0.13 is now slightly above the 1-stdev CI of the difference (~0.11). At N=200 the CI would shrink to ~0.08 and the Δ would be 1.6 standard errors above zero. Not formal statistical significance, but consistently directional.

ts_momentum is the harness’s load-bearing winner on single- signal data. The previous “baseline is the high bar” framing from PR #751’s session-summary v2 was wrong.

Finding 2: The three three_clock variants are statistically identical

three_clock_vol_weighted:    +0.799  stdev 1.210
three_clock_portfolio_vol:   +0.798  stdev 1.225
three_clock_momentum:        +0.798  stdev 1.232

Three arms, three different intervention shapes, identical means at N=100 (within rounding). The differences observed at N=10 — three_clock_vol_weighted “stacked above” the parents at +1.058 — were N=10 noise.

This is consistent with the +1.00 correlation found between three_clock_momentum and three_clock_portfolio_vol in PR #742’s matrix. They catch the same signal. The three_clock_vol_weighted variant differs slightly in mechanism (per-symbol vol normalisation) but ends up at the same Sharpe.

Implication: the three composite arms can be CONSOLIDATED. Three strategy classes, one effective signal. A future catalog cleanup could pick one as the canonical “three-clock composite” and deprecate the other two — saving operators from N+1 ways to express the same edge.

Finding 3: baseline ≈ spread_filter on this data

baseline:      +0.863  stdev 1.077  median +0.972  hit% 79%
spread_filter: +0.861  stdev 1.076  median +0.972  hit% 79%

Identical to 0.002 Sharpe; identical median; identical hit rate; identical min and max. The chop filter is dormant — the top-bottom spread rarely falls below the 4% threshold on this synthetic, so the filter never fires.

PR #742’s pairwise matrix showed baseline vs spread_filter at +1.00 correlation across seeds. The N=100 means confirm: same per-seed sequence → same descriptive stats.

This means: on this synthetic, spread_filter is a no-op. Operators picking between baseline and spread_filter are picking between identical strategies (or the spread_filter is so rarely firing that the small percentage difference is invisible at any seed count).

Finding 4: Multi-factor arms remain the worst

two_factor:        -0.666  (worst, hit% 25%)
three_factor:      -0.656  (hit% 26%)
four_factor_tuned: -0.614  (hit% 28%)
four_factor:       -0.556  (hit% 31%)

Confirms PR #759’s finding. The default mean-reversion-dominant weights (w_z = -1, w_s = -1, w_k = -0.5) flip the score direction against the data’s signal direction. Multi-factor strategies need data with mean-reversion alpha; this synthetic has momentum alpha.

The four-factor (kurtosis-included) variant is the LEAST-bad of the four — kurtosis penalty partially cancels the mean-reversion tilt. PR #608’s “higher moments add noise faster than signal” rule sharpens: kurtosis is less actively-harmful than skew when the underlying signal is momentum.

The refined ten-rule table

PR #751’s session-summary v2 had 10 discipline rules. The N=100 run adds nothing new but refines two:

# Rule (refined for N=100)
R1 Report all four (mean, stdev, min, max) at N ≥ 50
R6 Mean tightens 1/√N; min/max widen with N; at N=100, ts_momentum’s lead is 1.2 standard errors over baseline
R8 Intervention-point shifts must beat baseline at N ≥ 50 to be load-bearing claims

The other seven rules are unchanged.

The catalog cleanup question

If three_clock_momentum, three_clock_portfolio_vol, and three_clock_vol_weighted are statistically identical at N=100, should the catalog deprecate two of them?

Arguments for deprecation:

  • One canonical “three-clock” arm reduces ops surface.
  • New operators don’t waste time choosing between equivalents.
  • Tests covering 22 strategies are easier than 24.

Arguments against:

  • The variants differ in shape (score vs portfolio scale vs vol-normalisation) and might separate on REAL data even though they’re equivalent on synthetic.
  • The pairwise correlation is +1.00 ONLY when the gates are dormant; on real data with active vol regimes, the variants would diverge.
  • Deprecating shipped strategies sends a stale-thesis signal per ADR-0014.

The honest answer: the catalog should KEEP all three but ADD a note that they’re indistinguishable on this synthetic — the three’s value is differential on real data, not synthetic. The three-clock notes already make this point indirectly; the catalog could state it directly.

The closing observation

The N=100 run did three things:

  1. Confirmed ts_momentum’s lead.
  2. Identified three-clock variants as duplicates on this synthetic.
  3. Identified baseline ≈ spread_filter as duplicates.

None of these were visible at N=10. Two of them (#2, #3) were visible at N=50 but the convergence wasn’t yet tight. At N=100 the picture is clear.

The harness has converged on a stable ranking. The next required step is real data — until then, the synthetic ranking IS the truth we have, and ts_momentum leads.

deskgdfindgfstrategiesgsfeaturesgepromotionsgpapigabotgbwritinggw

Shortcuts are no-ops while typing in an input or textarea.