Six regression tests pin the findings
what you'll learn · Why some findings need regression tests while others don't, and how the six tests in the test_example_fomc_blackout_compare.py file form a coherent corpus.
The session's 100+ PRs produced 12+ discipline rules and 60+ research notes. Six of those findings now have CI-enforced regression tests pinning their empirical predictions against future drift. This note documents what each test catches and why the tests-vs-research-notes ratio matters.
The session has produced 12+ discipline rules. Most live in
research notes (the corpus is 60+ notes deep). Six findings —
the most load-bearing — also have CI-enforced regression tests
in tests/test_example_fomc_blackout_compare.py.
The six regression tests
| # | PR | Test name | What it pins |
|---|---|---|---|
| 1 | #782 | test_ts_momentum_leads_baseline_on_directional_data |
Entry-rule axis dominates (ts_momentum > baseline by Δ > 0.03 at N=30 single-signal) |
| 2 | #787 | test_multi_factor_strategies_negative_on_directional_data |
Default mean-reversion-dominant weights mismatch momentum signal (all three multi-factor variants mean < 0 at N=30) |
| 3 | #807 | test_vol_weighted_mean_revert_hedges_not_stacks |
Anti-correlated parents HEDGE not STACK (composite within ±0.1 of better parent at N=30 dual-signal) |
| 4 | #810 | test_baseline_spread_filter_equivalence_on_synthetic |
Dormant filter equivalence (|baseline - spread_filter| < 0.1 at N=30) |
| 5 | #818 | test_mom_ma_composite_stacks_above_baseline |
Redundant-measurement STACK (composite > baseline by Δ > 0.03 at N=30, same quantity at different smoothings) |
| 6 | #828 | test_vw_tc_composite_interferes_below_better_parent |
Multiplicative-interference INTERFERE (composite < better parent by Δ > 0.05 at N=30, different quantities) |
What each test catches
The tests don’t just verify “the code runs.” Each pins a QUALITATIVE finding that a future refactor could accidentally break. The breakage would either be:
- A real improvement (different default weights produce better-than-expected behaviour). Worth flagging for review.
- A bug (regression in a strategy class or feature).
- A change in the synthetic (data generation diverges from current; the test’s calibration is wrong).
The regression-test report lets a future operator distinguish “the result CHANGED” from “the result IS WRONG.”
The tests-vs-notes ratio
Research notes: 60+. Regression tests: 6.
The ratio is intentional. Notes capture EVERY observation worth keeping. Tests pin only the findings:
- Robust at N=30+ — small-N findings (see PR #784’s clustered-vol collapse) aren’t load-bearing enough.
- Quantifiable — qualitative claims (e.g., “the rule iterates four times”) can’t be CI-enforced.
- Asymmetric in importance — if the finding flipping direction would surprise the operator, it merits a test.
Most session findings are interesting OBSERVATIONS but not load-bearing CLAIMS. The 6 regression tests catch the most load-bearing claims.
The corpus as a coherent unit
The six tests aren’t independent — they form a coherent matrix covering:
| Axis | Tests covering it |
|---|---|
| Strategy shape > factor count | #782, #787 |
| Pairwise rule iterations | #807, #818, #828 |
| Synthetic-equivalence | #810 |
Adding a 7th test would target an uncovered axis:
- Cross-symbol intervention (PR #732/#734’s
portfolio_vol_gaterecovers vs per-symbol variants) - Sign-vs-rank conflict (PR #770’s
ts_filteredinterferes whilets_active_setrecovers)
Both are candidates; out of scope for this note.
Discipline rule
Research notes are cheap (one PR each). Regression tests are expensive (test code + maintenance + slower CI). Use notes for every finding worth noting; use regression tests only for findings that:
- Survive at N ≥ 30 seeds.
- Have a quantifiable directional prediction.
- Would surprise the operator if reversed.
The corpus has 12+ rules and 60+ notes; 6 of them clear all three filters. That ratio is approximately right.
What the next session inherits
A test corpus that runs in under 30 seconds. Six failures, each diagnostic. The next operator can:
- Add a 7th test when a new finding clears the filters.
- Run the existing 6 as a quick sanity check before any harness/strategy change.
- Use the failures’ messages as starting points for investigation when something breaks.
The tests are the platform’s vital signs. They don’t tell the operator what to DO; they tell the operator when something has CHANGED.
The closing observation
Six tests is a small number. The session shipped 100+ PRs; 1 in ~17 PRs produced a test-worthy finding. That’s roughly right — the ratio of “noise” (small-N suggestion, qualitative observation, single-seed result) to “signal” (robust directional prediction) is high in research.
The discipline of “test only what you’d want a future operator to notice” keeps the corpus tight. The corpus stays maintainable. The platform stays honest.