System Design Cases

MLOps Pipeline

MLOps Pipeline — end-to-end ML lifecycle: data validation, feature store materialization, training (Kubeflow/Metaflow), experiment tracking (MLflow/W&B), model registry, CI/CD for ML, canary/AB deployment via Istio router, prediction logging, drift monitoring (Evidently/Arize), business metric tracking, and automated retraining trigger. Includes ADRs on build-vs-managed (SageMaker/Vertex/Databricks vs self-host Kubeflow) and reproducibility (data+code+env+config+seed). Three scenarios: e2e pipeline happy-path, shadow mode + A/B test, drift-triggered auto-retrain.

2 мин чтения·reference · free·1 интерактивная диаграмма

MLOps: reproducible lineage, gated release, and observable rollback

A production ML pipeline coordinates data, code, training, evaluation, registry, serving, and monitoring. Automation can repeat mistakes quickly unless every artifact is pinned and promotion is gated by evidence.

Mental model

TFX describes reusable production components for data analysis, validation, training, model validation, and serving. Model the pipeline as gated components with lineage.
Hidden ML technical debt includes feedback loops, undeclared consumers, data dependencies, configuration, and world change. Include system and organizational failure paths.
NIST AI RMF organizes risk work across govern, map, measure, and manage functions. Tie release and incident processes to use-case risk, not a generic score.
The ML Test Score provides a rubric spanning data, model, infrastructure, and monitoring tests. Use multiple gates rather than one accuracy threshold.

Guarantees and boundaries

Every deployed model resolves to immutable lineage and approval evidence.
A failed mandatory gate prevents promotion.
Rollback targets a previously verified compatible bundle, not merely an older weight file.

Diagram scenarios

Reproducible training run. The run records immutable input and environment identities before producing a candidate.

Offline gate then bounded canary. Passing offline tests authorizes only a controlled release, not universal safety.

Rollback on regression. Online evidence can invalidate an apparently successful release.

Poisoned or biased feedback. Observed clicks and human corrections are delayed behavioral data, not automatically clean labels.

Architecture decision

Create immutable lineage for datasets, features, code, environment, parameters, and models. Validate data before training, evaluate candidates against the current champion and risk gates, deploy gradually with rollback, and treat online feedback as delayed, biased, and potentially poisoned.

Failure modes

Non-reproducible source data makes incident comparison impossible.
Offline improvement can regress latency, fairness, calibration, or online outcomes.
Automated feedback can amplify bias or accept poisoning.

Operational checklist

Pin data, labels, features, code, container, dependencies, configuration, and model digest.
Test pipeline restart and idempotency after partial failure.
Separate offline, shadow, canary, and full-release decisions.
Monitor data, system, prediction, and delayed outcome signals.

[CONCEPT]feature-store

Primary sources

MLOps Pipeline·MLOps Pipeline — end-to-end ML lifecycle: data validation, feature store materialization, training (Kubeflow/Metaflow), experiment tracking (MLflow/W&B), model registry, CI/CD for ML, canary/AB deployment via Istio router, prediction logging, drift monitoring (Evidently/Arize), business metric tracking, and automated retraining trigger. Includes ADRs on build-vs-managed (SageMaker/Vertex/Databricks vs self-host Kubeflow) and reproducibility (data+code+env+config+seed). Three scenarios: e2e pipeline happy-path, shadow mode + A/B test, drift-triggered auto-retrain.open in editor ↗

Концепты в этой главе1

Feature Store: ML feature management

Спроси про эту главу

стр. 137 / 198

← Предыдущая глава

Machine Learning Basics

Следующая глава →