System Design Cases
Windowing and Watermarks
Windowing & Watermarks concept page. Streaming pipeline: Kafka source feeds a Flink job (Watermark Assigner -> keyBy -> Window Operator -> RocksDB state). Window emissions go to an aggregates sink, late events go to a side-output sink. Four scenarios: tumbling 1m window aggregation, late event with allowedLateness=2m re-fires window, session window with gap=15min for user activity, plus an ADR contrasting event-time + watermark vs processing-time simplicity.
Event-time windows and watermarks: completeness estimates, lateness and updates
Window отвечает «где группировать event time», trigger — «когда emit», watermark — estimate input completeness, accumulation — «как update result». Смешивать эти четыре решения опасно.
Корректная модель
- Watermark is an estimate of event-time progress, not a hard completeness proof.
- Window assignment, trigger timing, allowed lateness and accumulation/update mode are independent.
- State cleanup happens after the declared lateness horizon, not necessarily at first output.
- Effective watermark can be held by slow/idle partitions; idleness changes the assumption explicitly.
Границы и компоненты
| Компонент | Ответственность |
|---|---|
| Active Partition A | Поставляет events and a local event-time watermark. |
| Late Partition B | Медленнее продвигает watermark and sends out-of-order events. |
| Idle Partition C | Не генерирует events; требует explicit idleness detection. |
| Watermark Coordinator | Комбинирует active input watermarks; не предсказывает невозможность late data. |
| Window State | Хранит keyed aggregates/events through allowed lateness. |
| Versioned Result Sink | Поддерживает upsert/retraction или explicit append revisions. |
| Too-Late Reconciliation | Принимает events after cleanup cutoff. |
Сценарии
On-time event-time firing
Events with timestamps in [10
,10) update the keyed window. When the effective watermark passes 10, the initial trigger emits a result but retains state for allowed lateness.Проверяемый исход: Initial output is version 1, not irreversible final truth.
Allowed late update
An event for the already-fired window arrives before the cleanup deadline. State still exists, so the aggregate changes and emits version 2/retraction.
Проверяемый исход: The sink deterministically replaces or revises v1; state was not deleted on first fire.
Idle partition handling
A min-style watermark stalls if one partition emits neither records nor watermarks. After a configured idleness timeout, the partition is excluded until it resumes, then late events follow policy.
Проверяемый исход: Idleness is an operational assumption, not evidence the source has no more old data.
Beyond allowed lateness
After watermark plus allowed lateness, window state is cleaned. A later event cannot be merged locally and is routed with reason/provenance for reconciliation.
Проверяемый исход: No silent discard and no false replay determinism promise when external/reference inputs changed.
Failure, concurrency и replay checklist
- Version output by window key and revision or support retractions.
- Preserve beyond-cutoff events for audit/reconciliation when business correctness requires.
- Checkpoint watermark/timer/state consistently.
- Make timestamp extraction, timezone and interval boundaries testable.
Формулы, units и допущения
- For active inputs, a conservative effective watermark is often min(local watermarks); one slow input dominates until idleness policy applies.
- Cleanup cutoff = window end + allowed lateness under the engine policy; output trigger may occur earlier.
- Window state ≈ key rate × window span × per-key/event state, multiplied by overlapping windows where applicable.
Числа выше — учебные inputs или размерностные формулы. Их нельзя выдавать за benchmark или SLA конкретного продукта.
Связанные темы
Первичные источники
- https://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf
- https://nightlies.apache.org/flink/flink-docs-stable/docs/concepts/time/
- https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/datastream/operators/windows/
- https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/datastream/event-time/generating_watermarks/
Scope note
Диаграмма показывает причинные границы и recovery contracts, а не скрытую реализацию конкретного managed-сервиса. Любая stronger guarantee действует только в явно названной transaction/checkpoint/acknowledgement boundary.