Feature stores: time-correct definitions across training and serving

A feature store can centralize definitions and retrieval, but it does not automatically prevent leakage, stale values, skew, or wrong entity joins. Correctness depends on event time, availability time, entity identity, transformations, and materialization state.

Mental model

• Feast documents historical feature retrieval using point-in-time joins to avoid future leakage. Make event and availability time first-class.
• Feast separates offline historical retrieval and online low-latency feature serving. Model separate stores and materialization state rather than magical consistency.
• Michelangelo describes shared feature and production ML platform components at Uber. Treat feature reuse as platform behavior with operational ownership.
• Hidden ML technical debt includes unstable data dependencies and undeclared consumers. Version definitions and track consumers before evolution.

Guarantees and boundaries

• Point-in-time joins use values available at the prediction cutoff under a documented time model.
• Online reads return value version and freshness state.
• No feature default is implicit.

Diagram scenarios

Point-in-time correct training set. Each example joins only feature values that were available at its prediction timestamp.

Online serving with freshness metadata. The service reads an entity-scoped value and decides explicitly whether its age is acceptable.

Late event and backfill correction. Late data can change offline history after an online decision has already occurred.

Missing feature and train-serve skew. An online miss or divergent transform must follow a tested fallback or fail policy.

Architecture decision

Version feature definitions and entity keys, build training sets with point-in-time joins using only data available at each prediction time, and expose freshness plus provenance online. Reuse transformation code where practical, but continuously compare offline and online values and define safe behavior for missing or stale features.

Failure modes

• Late events and backfills can change offline datasets after production decisions.
• Entity-key mismatch can join a plausible but wrong row.
• Offline and online transforms can drift without request failures.

Operational checklist

• Define event time, arrival time, prediction time, timezone, and tie-breaking.
• Version entities, feature definitions, transforms, and materialization checkpoints.
• Test missing, stale, duplicate, late, deleted, and backfilled data.
• Protect sensitive features with least privilege, retention, and audit.

Related concept

Primary sources

1. https://docs.feast.dev/getting-started/concepts/point-in-time-joins
2. https://docs.feast.dev/getting-started/concepts/feature-retrieval
3. https://www.uber.com/blog/michelangelo-machine-learning-platform/
4. https://proceedings.neurips.cc/paper_files/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html