Stream processing: time, state, replay and backpressure

Stream processing — непрерывное вычисление над unbounded input. Correctness определяется тем, что именно упорядочено, какой time domain используется, сколько state удерживается, откуда replay и как output переживает duplicate execution.

Корректная модель

• Ordering is scoped to a partition/key, not an entire distributed stream.
• Event time, ingestion time and processing time answer different questions.
• Replayable input plus checkpointed state still needs a cooperative sink for end-to-end guarantees.
• Backpressure controls rate; it does not create extra capacity or repair hot-key skew.

Границы и компоненты

Сценарии

Keyed ordering boundary

Events for one business key are routed to one ordered partition/operator key-group. There is no global order across keys unless added with expensive coordination.

Проверяемый исход: Per-key state transitions are deterministic under the stated partitioning and schema contract.

Out-of-order event time

An event arrives after newer event timestamps but before the effective watermark/allowed lateness cutoff. The operator updates the window and emits an update/retraction as the sink contract requires.

Проверяемый исход: Processing order is not confused with event-time order; corrections are explicit.

Failure replay and duplicate execution

The operator crashes after a sink call but before progress is durably checkpointed. Replay executes the input again. Stable event/output identity prevents a second logical effect.

Проверяемый исход: At-least-once execution is made safe at the output boundary; it is not renamed universal exactly once.

Backpressure and overload

Sink throughput falls below ingress. Backpressure propagates, lag grows and source retention risk rises. Scaling helps only if the hot key/state/sink can parallelize.

Проверяемый исход: Operations alerts before retention is exhausted and sheds/degrades optional work by policy.

Failure, concurrency и replay checklist

• Stable event IDs and schema versions at ingress.
• Bound state with semantically justified windows/TTL and monitor cleanup.
• Quarantine poison/schema-invalid events without blocking every partition forever.
• Test replay, rescale and upgrade from real checkpoint/savepoint artifacts.

Формулы, units и допущения

• Little-style state estimate: active keyed records ≈ arrival rate × retention time, adjusted for key skew and aggregations.
• Lag seconds is not merely message count: convert using recent partition-specific ingress rates and expose both.
• If sustained ingress > egress, backlog grows without bound until retention/storage/load-shed limit.

Числа выше — учебные inputs или размерностные формулы. Их нельзя выдавать за benchmark или SLA конкретного продукта.

Связанные темы

[CONCEPT]windowing-watermarks

[CONCEPT]apache-flink-deep

Первичные источники

• https://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf
• https://nightlies.apache.org/flink/flink-docs-stable/docs/concepts/time/
• https://nightlies.apache.org/flink/flink-docs-stable/docs/learn-flink/fault_tolerance/
• https://kafka.apache.org/41/design/design/

Scope note

Диаграмма показывает причинные границы и recovery contracts, а не скрытую реализацию конкретного managed-сервиса. Любая stronger guarantee действует только в явно названной transaction/checkpoint/acknowledgement boundary.