Fifty seeds reveal the tie
what you'll learn · Why the first composite-arm result to beat baseline at N=10 reverted to a tie at N=50, and what this confirms about the discipline rule of trusting only multi-seed-survivable claims.
At N=10, three_clock_vol_weighted beat baseline by +0.10 Sharpe — directionally encouraging but within the stdev (1.22). At N=50, the composite ties baseline at +0.765 — within rounding of baseline's +0.767. The directional signal didn't survive the higher seed count, exactly as the session-summary v2 note's caveat predicted.
PR #746’s headline result:
mean stdev max
baseline: +0.957 1.335 +2.879
three_clock_vol_weighted: +1.058 1.216 +3.347 ← above
The note (the-pairwise-rule-predicted-this-one)
called this out:
+0.101 vs baseline is well below stdev 1.216. Statistically the composite ties baseline at this seed count. The direction (above, not below) is the load-bearing observation; the magnitude needs more seeds.
PR #751’s session-summary v2 named the follow-up:
Higher seed count. 10 seeds isn’t enough to claim three_clock_vol_weighted beats baseline statistically. Re-run at N=50 or N=100.
This note documents the re-run.
The 50-seed result
arm mean stdev min max
vol_weighted +0.795 1.022 -1.030 +2.871
baseline +0.767 1.065 -1.091 +2.879
three_clock_vol_weighted +0.765 1.149 -2.337 +3.391
three_clock_momentum +0.726 1.215 -2.479 +3.290
The Δ between three_clock_vol_weighted and baseline:
−0.002. Within rounding. The 10-seed +0.10 reverted to a
tie.
vol_weighted (one of the composite’s parents) actually shows
the highest mean at N=50 (+0.795). At N=10 it was +0.930 — also
not a meaningful signal. The seed-by-seed variance dominates
the small differences between arms.
Two more findings the 50-seed run surfaces
The min Sharpe is much wider
N=10: three_clock_vol_weighted min -0.932
N=50: three_clock_vol_weighted min -2.337
The min Sharpe more than doubled. Higher seed count → more chances to draw a bad seed → wider observed worst-case. The N=10 min was a lucky observation; the N=50 min is closer to the true worst-case for the data distribution.
This sharpens ADR-0060 Rule 1 (rank by operator’s metric): operators using min Sharpe need MORE seeds, not fewer. The mean tightens slowly with seed count (1/√N); the min/max ENLARGES with seed count (because more draws = more extremes). A drawdown-constrained operator running at N=5 is reading a suspiciously-good worst-case.
The arm ordering shuffled
| Arm | N=10 rank | N=50 rank |
|---|---|---|
| baseline | 1 | 2 |
| vol_weighted | 2 | 1 |
| three_clock_momentum | 3 | 4 |
| three_clock_vol_weighted | 4 | 3 |
The “first composite to beat baseline” became the third-ranked arm at higher seed count. The rank flips are all within stdev, but they show how shaky a 10-seed ranking is when the true deltas are small.
What this confirms
-
The directional rule (composite stacks > parents) held but the magnitude (+0.10 → 0.00) didn’t survive. The qualitative direction of the pairwise-rule prediction remains correct; the quantitative magnitude was N=10 noise.
-
The variance discipline from PR #656 (
vol-regime-filter-mean-vs-variance) was right: ranking by mean alone is misleading when all arms cluster within stdev. The 50-seed run confirms baseline andthree_clock_vol_weightedare statistically equivalent.
The refined discipline rule
PR #710 / #742 / #748’s pairwise rule was about which composites will stack vs interfere. The N=50 result adds:
A composite that DIRECTIONALLY stacks (mean > both parents) may not statistically stack — the directional rule has sample-size cost. At N ≤ 20, +0.1 Sharpe differences are within noise; need N ≥ 50 to start distinguishing.
A corollary: before claiming a composite “works,” re-run at the highest seed count your time budget allows. The 1-second-per-seed harness can run N=100 in 100 seconds; the extra precision is worth the wait.
What this doesn’t show
-
Not that the composite is bad. Tying baseline at N=50 is a real result. It means the composite catches roughly the same total signal as baseline — same Sharpe at the same gross leverage. The operator’s choice between the two is about the variance shape (composite stdev 1.149 vs baseline 1.065 — slightly wider) or the worst-case (composite min −2.337 vs baseline −1.091 — composite is worse).
-
Not that the pairwise rule failed. The rule predicted directional stacking. At N=10 the data was consistent with the prediction; at N=50 it’s noisier. The rule’s cheap-pre-test property still holds — operators just need to set the bar at “directionally stacks” not “statistically stacks.”
-
Not that synthetic ≈ real. The real-data run is still the next required step. The N=50 synthetic result says “the composite ties baseline on synthetic”; real markets might produce a different rank order.
The next experiments
The session-summary v2’s three follow-ups remain:
-
Higher seed count for three_clock_vol_weighted← DONE here. Ties baseline at N=50. -
Lowest-correlation composite (
three_clock_vol_regime × ts_momentumat +0.27). Whether stacking continues into the under-0.3 band is unknown. This is now the highest-priority follow-up. -
Real-data run. Still required.