Search isn't research

There is a moment in every quant platform’s life where someone wires up “search the feature set” or “sweep the hyperparameters” and the loop never returns. The candidate count is a config knob, the loop is a for, and the spec just doesn’t have a ceiling. It worked on the laptop with 10 features. It runs for 14 hours on the cluster with 50.

This is the recurring failure mode of research platforms: search that became unbounded because nobody made it bounded. Not because the search algorithm is wrong — random sampling and greedy forward-selection are both fine — but because the spec that parameterises the search doesn’t carry a hard ceiling. The operator asks for “a sweep,” the platform delivers a sweep, and the cost shows up on next month’s invoice.

This post is about what bounded discovery looks like once you take the ceilings seriously, and what gets cheaper once you do.

The three methods

A research platform needs three search methods, and you can tell them apart by what they hold fixed:

• Factor mining. You don’t know what alpha signal to use. The search varies the expression — short symbolic combinations of primitives (mid_px / vwap, rsi_14 * vol_5) — and ranks by some scalar like |rank_ic|. Everything downstream of the expression is fixed.
• Feature search. You have a candidate pool of factors and a base model. The search varies the subset of features and holds the estimator + objective fixed. “Greedy forward selection until it stops improving” is the canonical shape.
• Sweep. You have features and an estimator. The search varies hyperparameters — window lengths, regularisation, the objective itself — and holds the rest fixed.

You’ll know the operator picked the wrong method by what they’re varying: a feature search over a factor that doesn’t exist yet should be factor mining; a sweep over features should be a feature search. Each method holds the other two methods’ axes fixed; if a research request seems to need to vary all three, the operator isn’t doing research, they’re doing exploration. Exploration is fine — but it should be costed differently.

The bound lives on the spec, not in the loop

The naïve place to bound discovery is in the loop:

for i, config in enumerate(generate_configs(...)):
    if i > 1000: break  # the ceiling
    ...

This is the wrong place. The operator doesn’t see the ceiling; if they bypass it (a one-line patch, a fork, a CLI flag) the loop runs forever again. The ceiling needs to live on the spec the operator writes:

@dataclass(frozen=True)
class SweepSpec:
    base: Model
    method: Literal["grid", "random"]
    n_iter: int                          # hard cap on random-search draws
    max_configs: int = 1000              # hard cap on grid product
    grid: dict[str, list[Any]]
    # constructor refuses if Π(grid.values) > max_configs

    def __post_init__(self) -> None:
        if self.method == "grid":
            size = math.prod(len(v) for v in self.grid.values())
            if size > self.max_configs:
                raise ValueError(f"grid would evaluate {size} configs; "
                                 f"max_configs={self.max_configs}")

The construction refuses; nothing runs. The operator either narrows the grid, raises the ceiling explicitly, or picks method="random" with a smaller n_iter. The choice is named in the spec; the cost is bounded before the first evaluation lands.

For an agent operator the bound is even more important — see the refusal-as-planning-hint note. When the spec construction refuses, the agent gets a typed slug it can plan around: try a smaller pool, try a different method, ask the human. An unbounded loop just stalls. An unbounded loop with no slug is a hallucinated checkpoint that pads tokens.

The HTTP surface is stricter than the CLI

The same discovery code runs in two contexts:

• The CLI / programmatic Python path. The operator is typing the spec. They see the ceiling. If they raise it, they’re consciously accepting a higher cost.
• The HTTP service (/v1/discovery/*). The caller is anyone the network can reach. They don’t see the ceiling source; they only see what the service is willing to serve.

These have different threat models, so they have different ceilings. The HTTP path materialises a synthetic L1 panel at request time and caps the run at five seconds of one worker’s CPU. That’s not enforcement of correctness — it’s enforcement of shared infrastructure. A request that wants more compute needs to be a CLI call against the operator’s own machine.

The corollary: the discovery service refuses runs that an HTTP caller could exploit to drain compute. It doesn’t refuse them in prose (“please use the CLI”); it refuses them with a typed slug (cross_product_too_large) the caller can branch on. The same typed-refusal contract the MCP tool surface uses for agents.

Discovery output isn’t a model

Here’s the part most platforms get wrong: a discovery run finishes, the leaderboard has a top entry, and someone operationalises that entry directly. Test failure follows shortly.

A discovery run produces a leaderboard, which is a ranking of candidates against a synthetic or partial evaluation. It is not a trained, registered, eligible-to-deploy model. To get from a leaderboard entry to a model that the runner can call:

1. The operator inspects the leaderboard.
2. The operator picks an entry and re-runs the full backtest with that entry’s config — not the cached score from the discovery run.
3. The full backtest produces an EvalReport.
4. ModelRegistry.register(eval_report, config) lands the model as a candidate.
5. The promotion gate (the-promotion-gate) makes it live, with a named human approver.

The discovery → leaderboard → backtest → register → promote chain is five steps for a reason: at every step the work narrows from “plausible” to “promotable.” Skipping a step is how a top leaderboard entry becomes a strategy that loses money for three weeks before the operator realises the discovery scoring used a contaminated panel. The five-step chain is the platform’s risk posture.

What gets cheaper

Once the spec carries the ceiling and the HTTP service refuses oversize requests by construction:

• Cost is predictable — the operator can describe a run as “this will evaluate at most 200 configs against 600 ticks of synthetic data” and the platform will refuse anything wider. Capacity planning becomes possible.
• Agent operators can plan — same surface, same typed refusals, same branching shape as for human operators. The agent reads cross_product_too_large and narrows its spec.
• The hand-off to promotion stays narrow — discovery produces candidates, not promotion-ready artifacts. The expensive, contaminated, oversold “I ran a search and now it’s live” path doesn’t exist because there’s no direct edge in the data model.
• The platform refuses to be a recipe book — bounded discovery is still discovery. The operator names what to vary; the platform refuses how much. The space between those two is where actual research happens.

What this isn’t

Bounded discovery isn’t grid search with a smaller grid. It isn’t “use random instead of exhaustive.” It’s the discipline of naming, in the spec the operator writes, the bound that makes a run finite — and refusing construction when the bound is missing or exceeded. The bound goes on the contract, not in the loop.

Once that’s in place, the search algorithm is almost uninteresting. Greedy works. Random works. The point isn’t the algorithm; the point is that unbounded search isn’t research — it’s a budget hole with a Jupyter notebook in front of it.

Bound the spec. Refuse the request. Let the operator be the one who decides what costs what.