System Design Cases
Stream Processing
Stream processing concept page (/concepts/stream-processing). Source -> operators -> sink pipeline. Stateless operators (filter, map) vs stateful (keyBy, tumbling window aggregate, RocksDB local state). Time semantics: event-time vs processing-time. Watermarks and late events with side output. Backpressure via credit-based flow control. Checkpoint-restore via Chandy-Lamport. Includes 5 scenarios (simple ETL, stateful windowed count, late event handling, backpressure cascade, checkpoint recovery) and 2 ADRs (stream-vs-batch-vs-lambda, event-time-vs-processing-time).
Stream processing: time, state, replay and backpressure
Stream processing — непрерывное вычисление над unbounded input. Correctness определяется тем, что именно упорядочено, какой time domain используется, сколько state удерживается, откуда replay и как output переживает duplicate execution.
Корректная модель
- Ordering is scoped to a partition/key, not an entire distributed stream.
- Event time, ingestion time and processing time answer different questions.
- Replayable input plus checkpointed state still needs a cooperative sink for end-to-end guarantees.
- Backpressure controls rate; it does not create extra capacity or repair hot-key skew.
Границы и компоненты
| Компонент | Ответственность |
|---|---|
| Event Producers | Создают stable event IDs, keys и event timestamps. |
| Partitioned Replay Log | Даёт per-partition order, retention и replay positions. |
| Keyed Stream Operator | Обрабатывает event-time logic и managed state. |
| Managed State and Checkpoints | Хранит recoverable state; size зависит от workload/serialization. |
| Idempotent Materialized Sink | Применяет event/version identity and exposes lag. |
| Late Event Side Output | Quarantine/reconciliation path beyond lateness policy. |
| Backpressure and Capacity Control | Наблюдает queues, lag, skew и overload policy. |
Сценарии
Keyed ordering boundary
Events for one business key are routed to one ordered partition/operator key-group. There is no global order across keys unless added with expensive coordination.
Проверяемый исход: Per-key state transitions are deterministic under the stated partitioning and schema contract.
Out-of-order event time
An event arrives after newer event timestamps but before the effective watermark/allowed lateness cutoff. The operator updates the window and emits an update/retraction as the sink contract requires.
Проверяемый исход: Processing order is not confused with event-time order; corrections are explicit.
Failure replay and duplicate execution
The operator crashes after a sink call but before progress is durably checkpointed. Replay executes the input again. Stable event/output identity prevents a second logical effect.
Проверяемый исход: At-least-once execution is made safe at the output boundary; it is not renamed universal exactly once.
Backpressure and overload
Sink throughput falls below ingress. Backpressure propagates, lag grows and source retention risk rises. Scaling helps only if the hot key/state/sink can parallelize.
Проверяемый исход: Operations alerts before retention is exhausted and sheds/degrades optional work by policy.
Failure, concurrency и replay checklist
- Stable event IDs and schema versions at ingress.
- Bound state with semantically justified windows/TTL and monitor cleanup.
- Quarantine poison/schema-invalid events without blocking every partition forever.
- Test replay, rescale and upgrade from real checkpoint/savepoint artifacts.
Формулы, units и допущения
- Little-style state estimate: active keyed records ≈ arrival rate × retention time, adjusted for key skew and aggregations.
- Lag seconds is not merely message count: convert using recent partition-specific ingress rates and expose both.
- If sustained ingress > egress, backlog grows without bound until retention/storage/load-shed limit.
Числа выше — учебные inputs или размерностные формулы. Их нельзя выдавать за benchmark или SLA конкретного продукта.
Связанные темы
Первичные источники
- https://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf
- https://nightlies.apache.org/flink/flink-docs-stable/docs/concepts/time/
- https://nightlies.apache.org/flink/flink-docs-stable/docs/learn-flink/fault_tolerance/
- https://kafka.apache.org/41/design/design/
Scope note
Диаграмма показывает причинные границы и recovery contracts, а не скрытую реализацию конкретного managed-сервиса. Любая stronger guarantee действует только в явно названной transaction/checkpoint/acknowledgement boundary.