System Design Cases

Feature Store: ML feature management

Feature Store: centralized ML feature management with online + offline serving. Solves training-serving skew via single feature definition synced to two stores. Tools: Feast (OSS standard), Tecton (managed streaming-first), Hopsworks, Vertex Feature Store. Online store (Redis/DynamoDB) for low-latency model inference. Offline store (S3/Parquet) for training with point-in-time joins. Materialization engine syncs both stores from batch (Spark) and streaming (Flink) compute. Includes ADR considerations for build vs Feast vs managed Tecton.

2 мин чтения·reference · free·1 интерактивная диаграмма

Feature stores: time-correct definitions across training and serving

A feature store can centralize definitions and retrieval, but it does not automatically prevent leakage, stale values, skew, or wrong entity joins. Correctness depends on event time, availability time, entity identity, transformations, and materialization state.

Mental model

Feast documents historical feature retrieval using point-in-time joins to avoid future leakage. Make event and availability time first-class.
Feast separates offline historical retrieval and online low-latency feature serving. Model separate stores and materialization state rather than magical consistency.
Michelangelo describes shared feature and production ML platform components at Uber. Treat feature reuse as platform behavior with operational ownership.
Hidden ML technical debt includes unstable data dependencies and undeclared consumers. Version definitions and track consumers before evolution.

Guarantees and boundaries

Point-in-time joins use values available at the prediction cutoff under a documented time model.
Online reads return value version and freshness state.
No feature default is implicit.

Diagram scenarios

Point-in-time correct training set. Each example joins only feature values that were available at its prediction timestamp.

Online serving with freshness metadata. The service reads an entity-scoped value and decides explicitly whether its age is acceptable.

Late event and backfill correction. Late data can change offline history after an online decision has already occurred.

Missing feature and train-serve skew. An online miss or divergent transform must follow a tested fallback or fail policy.

Architecture decision

Version feature definitions and entity keys, build training sets with point-in-time joins using only data available at each prediction time, and expose freshness plus provenance online. Reuse transformation code where practical, but continuously compare offline and online values and define safe behavior for missing or stale features.

Failure modes

Late events and backfills can change offline datasets after production decisions.
Entity-key mismatch can join a plausible but wrong row.
Offline and online transforms can drift without request failures.

Operational checklist

Define event time, arrival time, prediction time, timezone, and tie-breaking.
Version entities, feature definitions, transforms, and materialization checkpoints.
Test missing, stale, duplicate, late, deleted, and backfilled data.
Protect sensitive features with least privilege, retention, and audit.

[CONCEPT]model-serving

Primary sources

Feature Store: ML feature management·Feature Store: centralized ML feature management with online + offline serving. Solves training-serving skew via single feature definition synced to two stores. Tools: Feast (OSS standard), Tecton (managed streaming-first), Hopsworks, Vertex Feature Store. Online store (Redis/DynamoDB) for low-latency model inference. Offline store (S3/Parquet) for training with point-in-time joins. Materialization engine syncs both stores from batch (Spark) and streaming (Flink) compute. Includes ADR considerations for build vs Feast vs managed Tecton.open in editor ↗

Концепты в этой главе1

Model Serving

Спроси про эту главу

стр. 138 / 198

← Предыдущая глава

MLOps Pipeline

Следующая глава →