Trading data engineering: what breaks strategies before the market does

Backtests fail quietly

A flawed backtest does not raise an error. It returns a number, and the number is usually flattering. That asymmetry is why market data pipelines deserve the same rigour applied to laboratory data: explicit rules, recorded provenance, and checks that run before results are trusted.

The five failures worth engineering against

Across market data work, the same small set of issues accounts for most of the gap between simulated and realised performance.

• Timestamp ambiguity — exchange time versus receipt time versus storage time, mixed across venues and daylight-saving boundaries
• Look-ahead leakage — fundamentals, restatements or index membership applied earlier than they were actually knowable
• Corporate actions — splits, dividends and symbol changes applied inconsistently across price and volume series
• Survivorship — universes built from instruments that still exist, quietly deleting the failures
• Silent gaps — missing bars, stale quotes and halted sessions treated as valid observations

Point-in-time or nothing

The single most valuable property of a trading data store is that any historical query can be answered as of a chosen moment. That requires storing when a fact was observed, not only what it says — a knowledge timestamp alongside the event timestamp.

With that in place, leakage becomes structurally difficult rather than a matter of discipline. Without it, every researcher is one careless join away from an unpublishable result.

A validation layer that runs on every load

The same rule-based validation used for analytical chemistry transfers directly to market data: expected bar counts per session, monotonic timestamps, price continuity thresholds, volume plausibility, cross-venue reconciliation, and adjustment-factor consistency.

Each check has a verdict, a reason and a record. Failures quarantine the load rather than contaminating the research set.

From research to production without a rewrite

Research code and production code should share the same data access layer, the same adjustment logic and the same validation rules. When they diverge, the strategy that was tested is not the strategy that trades.

The architecture that avoids this is unglamorous: one canonical store, one transformation library, configuration instead of forks, and structured logging at every boundary. It is also what makes a system reviewable by a risk or compliance function — which is increasingly not optional.