The promotion gate: why bad data should be unreachable
what you'll learn · Why structural-impossibility (a gate) beats human-reaction (a dashboard) for data validation.
Validation that runs as a dashboard tells you what went wrong after the model trained. Validation that runs as a gate makes it structurally impossible for a strategy to read what the platform hasn't vouched for. The difference shows up in PnL.
Most validation frameworks I’ve seen are dashboards: they compute checks over data already published, surface failures in a UI, and trust humans to act on them. This is the wrong default. By the time a check fails on Monday’s data, a research engineer has already loaded it into a notebook, a model has trained on it, and the resulting signal has been promoted upstream. The dashboard is a record of failures that already happened.
The right default is a gate: validation runs at a boundary, and data that fails the gate is structurally unreachable downstream.
Two namespaces, one boundary
The structural change is small. The lake has two namespaces.
raw.<vendor>.<exchange>.<type> holds what arrived from the producer.
Schema-valid but not vetted. Writable by the normalize pipeline,
readable by the validation framework and by incident-response tools.
published.<exchange>.<type> holds the same data, validated.
Readable by consumers — backtest clients, feature pipelines, strategy
containers. Not directly writable by anyone. It populates only by
promotion from raw after the gate passes.
The promotion runs as part of the normalize DAG. The validator reads
the candidate raw partition, applies the spec for that data type,
and either promotes (copy-on-write or Iceberg snapshot reference into
published) or refuses and quarantines.
What that buys
The argument isn’t “fewer bugs.” The argument is PnL drift.
A strategy that backtests against data the platform vouched for, and
trades live against data the platform vouched for, observes one
source of drift: the strategy itself. A strategy that backtests
against published but lives off of raw S3 reads — or backtests
against partially-validated data because the dashboard only fires
once a day — observes drift from many sources, and the human can’t
tell which one mattered after the fact. The most expensive class of
drift is the kind that backtest cannot reproduce because the bad
input is no longer there.
The promotion gate makes this class of drift impossible. It is not the only thing that needs to be true for the platform to be honest, but it is the single change with the highest leverage on backtest-vs-live PnL drift.
What it costs
The gate is a single task in the per-vendor DAG. Cost of building it: ~one week if the validation framework already exists, ~three weeks if it doesn’t. Cost of running it: the normalize wall-clock plus the spec evaluation, typically tens of seconds per session per symbol.
The harder cost is the discipline of refusing to bypass it.
Researchers will ask for direct raw access at times when the gate
is mid-tuning and rejecting too aggressively. Production-support
engineers will be tempted to fix data by hand and skip the gate. The
discipline is “no, fix the spec, fix the producer, widen the
threshold for one day with an audit trail.” Without that, the gate
erodes into a dashboard again within six months.
What it doesn’t solve
The gate enforces the platform’s view of data quality. It does not enforce that the spec is correct. A spec that’s too lax lets bad data through; a spec that’s too strict blocks good data and erodes trust. Tuning is real work, sometimes ongoing.
The gate also does not enforce the consumer’s understanding of the data. A spec that says “spread > 0” passes a session where every quote has a one-cent spread but the underlying volatility was zero; the model trained on it will overfit. Validation gates what the producer ships; what the consumer assumes is still the consumer’s problem.
The discipline behind the structure
A team can have validation as a dashboard or validation as a gate. The hardware cost is similar. The decision is whether the team is willing to make “this data is not yet readable” a real, daily condition of the lake. Below that willingness, validation never quite makes it into the read path, and the same incidents recur.
The gate is not a checkbox. It is a posture.