Right-sizing the research data platform: five thresholds, in order

There are two architectures for a research data platform, both correct, neither universal. A team picks the wrong one by default: either over-builds at one person and dies under coordination overhead, or under-builds at twenty and dies under drift. The decision should be made by which threshold the team has crossed, and threshold-crossings are observable.

The two architectures

Minimal. One repo, Python, hive-partitioned parquet on S3, DuckDB and Polars as the read path, cron + a Slack webhook as the orchestrator, AWS credentials in a .env file. ~$200/mo to run, one to two weeks to build, ~1,500 lines of code.

Firm-scale. Seven repos including a schemas/ source with cross-language codegen, Iceberg + Nessie storage with a raw → published promotion gate, two clients with identical chaining APIs (live and batch), a FastAPI catalog UI, a strategy template that bakes in the lake client, OIDC SSO, Airflow with on-failure callbacks. ~$15k/mo to run, two engineers for ninety days to build.

Both architectures generate alpha when the strategies inside them are good. Both fail to generate alpha when the strategies are bad. The architecture is not the trade.

What the bigger one buys, in trading-outcome terms

Five capabilities that move the trading book — not lists of features, but the consequences of those features at the strategy level.

Validation as a gate. Strategies trade against data the platform vouched for. Bad vendor days are quarantined in a raw namespace; consumers only see published. The cost saved is the cost of one strategy training on a crossed-book day before the human notices.

Live-trading reliability. SLAs, on-call alerts, kill switches, failover. Required the moment real capital is deployed; absent below.

Reproducibility. Time-travel on the lake, content-hash versioning on features, schema codegen across languages. The question “what data was the model trained on?” must be answerable six months later, not in principle but in practice.

Multi-person coordination. Catalog, schema source-of-truth, SSO. At one person, overhead. At twenty, the difference between linear and quadratic coordination cost.

Strategy throughput at scale. Feature reuse, fast vendor onboarding, parallel research without conflict. Firm-level Sharpe scales with strategy count more than with per-strategy quality — assuming you can actually run them in parallel.

Worth saying explicitly: this list does not contain “alpha.” The platform doesn’t generate alpha; it lets people pursue it without infrastructure-caused drift.

What the smaller one preserves

Five things that bigger platforms erode.

Per-researcher iteration speed (no codegen step before a rerun). Cognitive load of order O(1) — everything fits in one head. Cash cost low enough to iterate on personal money. Zero meeting overhead. Total ownership clarity: every bug is yours, every fix lands immediately. Below ~five people, none of these are luxuries; they’re the working conditions that produce hits.

The five thresholds

The right architecture is the smallest one that covers the currently-crossed thresholds. Cross any of these and the minimal stack starts losing more than it saves.

1. Live capital deployed. Below this, you can absorb a silent bad-data day. Above it, the cost of one such day exceeds the lifetime cost of building the gate.
2. Headcount above ~5. Shared conventions in one repo break when the next hire can’t cd into your laptop. You need a discoverable catalog and a schema source of truth from this point.
3. Strategy count above ~10. Feature reuse becomes mandatory. Two researchers computing mid_price = (bid + ask) / 2 slightly differently is fine at two strategies; it’s silent drift at twenty.
4. Regulated markets. Equities or futures in major jurisdictions makes audit trail a legal requirement, not a nice-to-have.
5. Multi-timezone operations. The “cron and pray” model breaks the moment you can’t reach the researcher who would fix the 3am failure.

Below all five: minimal is strictly better. Above any one: upgrade the corresponding piece.

The partial upgrade order

The two architectures aren’t binary; you absorb pieces in order as thresholds land.

1. Validation as a gate (not inline asserts) when live capital is deployed. ~one week.
2. Iceberg (still no Nessie) when reproducibility becomes legal or scale-driven. ~two weeks.
3. Catalog UI when ~five people stop knowing what data exists. ~one week — FastAPI scanning the lake metadata, not a full Nessie setup yet.
4. Schema codegen when languages cross (Python + Rust + Java). ~two weeks.
5. Feature store when researchers duplicate features. ~two weeks.
6. OIDC SSO, Nessie, Kubernetes, Airflow when you have a platform team to operate them. ~two months.

Each step costs ~one to two weeks and pays back if the corresponding threshold has been crossed. Adding them before is premature, and the cost is real — every researcher who learned the old way has to relearn the new one. Adding them after is also real, and that cost is drift.

The discipline that’s universal

Some practices belong at both scales, and they are the ones that survive any restructuring.

One schema source. A file, a module, a repo — pick one — that every loader imports from. String-literal column names duplicated across three notebooks is the failure mode regardless of headcount.

Validation asserts inline. Five lines per loader, before the DataFrame returns. Catches 90% of the issues that otherwise surface as model misbehavior at hour three of a backtest.

Symbol identity. Cache the venue’s catalog (ccxt.load_markets() for crypto, OpenFIGI for equities) to a parquet nightly. Even solo, you need to know which BTCUSDT is which exchange.

PnL drift measurement. Backtest-vs-live drift on a fixed strategy is the only honest test of the platform. Solo or thirty-person, the question is the same: does the backtest match live?

The closing test

Engineering proxies — schema-rename PR counts, build times, test coverage — matter only insofar as they correlate with three numbers visible at the book level.

• T1, backtest-vs-live PnL drift. Below one bps/day on a fixed strategy: the lake reproduces what live trading sees. Above five: something in the pipeline is lying.
• T2, idea to paper-trade. New feature on existing data in under a week; new vendor onboarded in under three. Strategy throughput compounds. This is the dimension where small architectures shine and large ones get expensive.
• T3, bad-data detect time. Producer ships a crossed book; how long until consumers are protected? Under an hour at firm-scale, under a day at minimal. Beyond that, downstream models are training on noise and you’ve already paid.

Build the smallest architecture that puts these three numbers in defensible territory for the size you actually are. Reassess every six months. The instinct to “build the right thing from the start” is the instinct to build the firm-scale architecture before headcount or AUM justify it; that instinct is wrong. The opposite instinct — to never upgrade past a comfortable size — is how a five-person team accretes coordination overhead until it is operating a firm-scale platform with one person’s discipline. The threshold-crossings are the only honest signal.