How to use the comparison harness

The harness’s surface area grew substantially this session. This note is the operator’s how-to: a step-by-step workflow for asking “does my new strategy work?” and reading the answer.

Phase 1: Add the strategy

A new strategy lives in alphakernel/exec/strategies.py. The shape:

class MyNewStrategy:
    slug: str = "my_new"
    cites: tuple[str, ...] = ("walk-forward-without-leakage",)
    feature_slugs: tuple[str, ...] = ("my_new_lookback",)

    def __init__(self, *, tickers, lookback=20, ...):
        # Validate. Build per-symbol history deques.
        ...

    @property
    def lookback(self) -> int:
        return self._lookback

    def compute_weights(self, latest_prices):
        self.observe(latest_prices)
        # Compute per-symbol scores; pass to _xs_weights_from_scores
        # for the cross-sectional ranking + leg-split.
        ...

    def observe(self, latest_prices):
        # Keep deques warm during ticks compute_weights returns {}.
        ...

Add unit tests in tests/test_strategies.py for:

1. Cold-start returns {}.
2. Invariants the strategy claims (sign convention, threshold behaviour, etc.).
3. All ValueError validation paths.
4. observe keeps history warm without compute_weights being called.

Per ADR-0058, if the strategy is event-aware, include a test that clears cited_event_artifacts on cold inner.

Phase 2: Add the arm

Edit examples/fomc_blackout_compare.py:

1. Import the new strategy class.
2. Inside _run_one_seed, instantiate it with the harness’s common params (tickers, lookback, long_quantile, gross_leverage) and walk it through _walk.
3. Add the result to out dict with a short slug.
4. Add the print-row in single-seed mode.
5. Add to the multi-seed sharpe_by_arm dict + the arms tuple.
6. Add to _print_arm_list() for --list-arms discoverability.
7. Add to the test file’s expected-arms list.

Phase 3: Run the harness

# Quick sanity check (single seed, 100 days).
python examples/fomc_blackout_compare.py --seed 7 --days 100

# Multi-seed dispersion (recommended for any claim).
python examples/fomc_blackout_compare.py \
    --n-seeds 5 --days 200 --fomc-drift-bps 50

# Dual-signal mode (composites stack rather than interfere).
python examples/fomc_blackout_compare.py \
    --n-seeds 5 --fomc-drift-bps 50 --mean-revert-bps 100

# Sort by your constraint.
python examples/fomc_blackout_compare.py \
    --n-seeds 5 --fomc-drift-bps 50 --sort-by mean
python examples/fomc_blackout_compare.py \
    --n-seeds 5 --fomc-drift-bps 50 --sort-by min   # worst-case
python examples/fomc_blackout_compare.py \
    --n-seeds 5 --fomc-drift-bps 50 --sort-by stdev # most stable

# Discoverability — no backtest, just the arm inventory.
python examples/fomc_blackout_compare.py --list-arms

Phase 4: Export + analyse

# Save per-(seed, arm) rows for ad-hoc analysis.
python examples/fomc_blackout_compare.py \
    --n-seeds 20 --fomc-drift-bps 50 \
    --save-csv /tmp/harness.csv

# Percentile + hit-rate stats the harness doesn't print.
python scripts/analyse_harness_csv.py /tmp/harness.csv --top 8

# Pairwise correlation across arms (predicts composition outcome).
python scripts/analyse_harness_csv.py /tmp/harness.csv \
    --top 8 --pairwise

The CSV is small (19 arms × N seeds × 5 floats) and survives across runs — operators can load it into pandas / a notebook for further analysis without re-running the harness.

Phase 5: Read the answer

Per ADR-0060’s three rules:

Rule 1: Rank by your constraint, not mine

Operator situation	Rank by	Why
Many independent strategies	--sort-by mean	Portfolio averages
One strategy, must ride a year	--sort-by min	Worst-case is the gate
Drawdown-constrained	--sort-by min	Realisable lower bound
Tactical overlay, weekly review	--sort-by mean	Can cut bad regimes fast
Care about consistency	--sort-by stdev	Lowest dispersion = most predictable

The hit_rate column from analyse_harness_csv.py is the operator-grade gate for one-strategy deployment: fraction of seeds with positive Sharpe. Below 80% → not a candidate for single-deployment.

Rule 2: Sharpe comparisons need equal-leverage controls

Don’t compare two strategies at different gross_leverage. The harness’s equal_risk_long_only arm verifies this: long-only at 0.71× gross has identical Sharpe to long-only at 1.0× gross. Scale changes dollar outcomes, not Sharpe.

Rule 3: Run the composite before declaring two top arms

If two arms tie at the top of the mean-Sharpe ranking, build a composite arm (gate ∘ score) and run it. The result distinguishes:

• Stack: composite mean > both parents. Different mechanisms; composing is sound.
• Interfere: composite mean between parents OR min worse than both. Same mechanism; can’t double-count.

Don’t ship the composite arm to production — it’s a measurement tool. Production gets the higher-mean parent OR the higher-min parent, by Rule 1.

Pre-test: before building the composite arm, look at the pairwise correlation matrix (pairwise-correlation-predicts-composition). Correlation > 0.8 → composite will interfere; < 0.3 → composite will stack. Saves the composite arm code if the prediction is clear.

What this workflow doesn’t cover

• Real-data deployment. ADR-0014 says claims need real-data verification. The harness produces scaffold evidence; the real-data run is the next step. Point this at SP500 + Polygon daily bars + the live FOMC calendar; the same toolchain works.

• Sensitivity analysis. All harness defaults (lookback, long_quantile, gross_leverage, threshold) are tunable but not varied automatically. A future operator wanting a heatmap across (lookback, threshold) needs to script it on top of _run_one_seed.

• Cross-data-source comparison. The harness uses one synthetic generator. Comparing single-signal vs dual-signal modes (per this session) is the closest analog; a fuller data-mode sweep would need additional generators.

The discipline rule

When asking “does this strategy work?”, the harness’s answer is multi-dimensional. Run multi-seed. Read all four stats. Pick the metric your deployment requires. Use pairwise correlation to pre-test composites. Don’t commit to “it works” on a single seed’s Sharpe.

A corollary: the workflow takes ~30 seconds per harness run at the defaults (10 symbols, 200 days, 5 seeds). The cost of doing each step is small enough that an operator should run them all before claiming a result — there’s no “fast path.”