Kappa and Lambda architectures: replay and serving trade-offs

Lambda и Kappa — patterns для recomputation, а не products и не гарантии. Lambda платит за две processing paths; Kappa переносит сложность в retention, deterministic replay, schema evolution и output replacement.

Корректная модель

• Lambda has separate batch and speed computations plus serving reconciliation.
• Kappa uses a retained log and replays a versioned stream topology; retention is configured, not assumed.
• Mutable reference data must be versioned/as-of for deterministic historical recomputation.
• Rebuild into a new output epoch and promote atomically instead of in-place corruption.

Границы и компоненты

Сценарии

Lambda dual processing paths

Speed output is provisional until a batch recomputation for the same logical time supersedes or reconciles it. Different code paths can diverge.

Проверяемый исход: Serving records carry source/version provenance so conflict is detected rather than silently merged.

Lambda reconciliation

Batch result differs from the provisional speed result because of late data or code drift. A named rule promotes the complete epoch and retires the provisional range.

Проверяемый исход: No request combines incompatible rows from two epochs.

Kappa versioned replay

A new topology reads retained history from a chosen offset into a green output while the old topology continues serving blue.

Проверяемый исход: Replay does not overwrite live output in place; a verified epoch switch makes it visible.

Retention and side-effect gap

Kappa cannot replay data that retention removed, and neither pattern can safely repeat an unkeyed external effect. Recovery falls back to archive/backfill or reconciliation.

Проверяемый исход: Architecture documentation names non-replayable dependencies and keeps an archive or authoritative snapshot where required.

Failure, concurrency и replay checklist

• Pin code, schema and reference-data versions for replay.
• Keep immutable history/archive long enough for required correction horizon.
• Make serving writes idempotent by event/output version.
• Inventory non-replayable external effects and reconcile them separately.

Формулы, units и допущения

• Replay completion time ≈ retained records / sustainable replay records/s, while live tail continues to grow.
• Lambda cost includes two compute paths plus reconciliation/serving complexity; no universal percentage applies.
• Kafka retention defaults are configuration/version properties and are never taught as a fixed 30 days.

Числа выше — учебные inputs или размерностные формулы. Их нельзя выдавать за benchmark или SLA конкретного продукта.

Связанные темы

[CONCEPT]stream-processing

[CONCEPT]change-data-capture

Первичные источники

• https://www.oreilly.com/radar/questioning-the-lambda-architecture/
• https://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf
• https://kafka.apache.org/41/configuration/topic-configs/
• https://kafka.apache.org/41/design/design/

Scope note

Диаграмма показывает причинные границы и recovery contracts, а не скрытую реализацию конкретного managed-сервиса. Любая stronger guarantee действует только в явно названной transaction/checkpoint/acknowledgement boundary.