System Design Cases
Lambda vs Kappa Architecture
Lambda vs Kappa Architecture concept page. Lambda (Marz 2011): three layers — batch (Spark, точно/медленно) + speed (Storm/Flink, быстро/приближённо) + serving (merge views). Kappa (Kreps 2014): один streaming pipeline через Kafka log + Flink EOS, replay через blue-green datasource swap (new consumer group offset=earliest, отдельный output namespace, atomic swap queries). Lakehouse hybrid (2026): Iceberg/Delta как unified primitive, Spark Structured Streaming + batch backfill пишут в одну table, Materialize держит incremental view. Сценарии: Lambda dual pipeline, Kappa happy path, Kappa replay, Lakehouse hybrid, Lambda merge bug failure mode. ADR-001: когда Kappa default, когда Lambda оправдан. ADR-002: blue-green datasource swap для replay.
Kappa and Lambda architectures: replay and serving trade-offs
Lambda и Kappa — patterns для recomputation, а не products и не гарантии. Lambda платит за две processing paths; Kappa переносит сложность в retention, deterministic replay, schema evolution и output replacement.
Корректная модель
- Lambda has separate batch and speed computations plus serving reconciliation.
- Kappa uses a retained log and replays a versioned stream topology; retention is configured, not assumed.
- Mutable reference data must be versioned/as-of for deterministic historical recomputation.
- Rebuild into a new output epoch and promote atomically instead of in-place corruption.
Границы и компоненты
| Компонент | Ответственность |
|---|---|
| Canonical Event History | Authoritative immutable/versioned facts with explicit retention. |
| Batch Recompute Layer | Строит complete views from bounded snapshots/history. |
| Speed Layer | Даёт low-latency provisional updates. |
| Versioned Stream Topology | Kappa processing code that supports controlled replay. |
| Versioned Reference Data | Makes historical replay reproducible instead of joining current mutable state. |
| Serving View Blue | Current visible materialization. |
| Serving View Green | Rebuilt/replayed candidate promoted by epoch. |
| Serving Epoch Router | Atomically selects one complete output version. |
Сценарии
Lambda dual processing paths
Speed output is provisional until a batch recomputation for the same logical time supersedes or reconciles it. Different code paths can diverge.
Проверяемый исход: Serving records carry source/version provenance so conflict is detected rather than silently merged.
Lambda reconciliation
Batch result differs from the provisional speed result because of late data or code drift. A named rule promotes the complete epoch and retires the provisional range.
Проверяемый исход: No request combines incompatible rows from two epochs.
Kappa versioned replay
A new topology reads retained history from a chosen offset into a green output while the old topology continues serving blue.
Проверяемый исход: Replay does not overwrite live output in place; a verified epoch switch makes it visible.
Retention and side-effect gap
Kappa cannot replay data that retention removed, and neither pattern can safely repeat an unkeyed external effect. Recovery falls back to archive/backfill or reconciliation.
Проверяемый исход: Architecture documentation names non-replayable dependencies and keeps an archive or authoritative snapshot where required.
Failure, concurrency и replay checklist
- Pin code, schema and reference-data versions for replay.
- Keep immutable history/archive long enough for required correction horizon.
- Make serving writes idempotent by event/output version.
- Inventory non-replayable external effects and reconcile them separately.
Формулы, units и допущения
- Replay completion time ≈ retained records / sustainable replay records/s, while live tail continues to grow.
- Lambda cost includes two compute paths plus reconciliation/serving complexity; no universal percentage applies.
- Kafka retention defaults are configuration/version properties and are never taught as a fixed 30 days.
Числа выше — учебные inputs или размерностные формулы. Их нельзя выдавать за benchmark или SLA конкретного продукта.
Связанные темы
Первичные источники
- https://www.oreilly.com/radar/questioning-the-lambda-architecture/
- https://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf
- https://kafka.apache.org/41/configuration/topic-configs/
- https://kafka.apache.org/41/design/design/
Scope note
Диаграмма показывает причинные границы и recovery contracts, а не скрытую реализацию конкретного managed-сервиса. Любая stronger guarantee действует только в явно названной transaction/checkpoint/acknowledgement boundary.