Event-time windows and watermarks: completeness estimates, lateness and updates

Window отвечает «где группировать event time», trigger — «когда emit», watermark — estimate input completeness, accumulation — «как update result». Смешивать эти четыре решения опасно.

Корректная модель

• Watermark is an estimate of event-time progress, not a hard completeness proof.
• Window assignment, trigger timing, allowed lateness and accumulation/update mode are independent.
• State cleanup happens after the declared lateness horizon, not necessarily at first output.
• Effective watermark can be held by slow/idle partitions; idleness changes the assumption explicitly.

Границы и компоненты

Компонент	Ответственность
Active Partition A	Поставляет events and a local event-time watermark.
Late Partition B	Медленнее продвигает watermark and sends out-of-order events.
Idle Partition C	Не генерирует events; требует explicit idleness detection.
Watermark Coordinator	Комбинирует active input watermarks; не предсказывает невозможность late data.
Window State	Хранит keyed aggregates/events through allowed lateness.
Versioned Result Sink	Поддерживает upsert/retraction или explicit append revisions.
Too-Late Reconciliation	Принимает events after cleanup cutoff.

Сценарии

▶Walkthrough: on-time-window-fire (not found)

On-time event-time firing

Events with timestamps in [10

,10) update the keyed window. When the effective watermark passes 10, the initial trigger emits a result but retains state for allowed lateness.

Проверяемый исход: Initial output is version 1, not irreversible final truth.

▶Walkthrough: allowed-late-update (not found)

Allowed late update

An event for the already-fired window arrives before the cleanup deadline. State still exists, so the aggregate changes and emits version 2/retraction.

Проверяемый исход: The sink deterministically replaces or revises v1; state was not deleted on first fire.

▶Walkthrough: idle-partition-watermark (not found)

Idle partition handling

A min-style watermark stalls if one partition emits neither records nor watermarks. After a configured idleness timeout, the partition is excluded until it resumes, then late events follow policy.

Проверяемый исход: Idleness is an operational assumption, not evidence the source has no more old data.

▶Walkthrough: too-late-reconciliation (not found)

Beyond allowed lateness

After watermark plus allowed lateness, window state is cleaned. A later event cannot be merged locally and is routed with reason/provenance for reconciliation.

Проверяемый исход: No silent discard and no false replay determinism promise when external/reference inputs changed.

Failure, concurrency и replay checklist

• Version output by window key and revision or support retractions.
• Preserve beyond-cutoff events for audit/reconciliation when business correctness requires.
• Checkpoint watermark/timer/state consistently.
• Make timestamp extraction, timezone and interval boundaries testable.

Формулы, units и допущения

• For active inputs, a conservative effective watermark is often min(local watermarks); one slow input dominates until idleness policy applies.
• Cleanup cutoff = window end + allowed lateness under the engine policy; output trigger may occur earlier.
• Window state ≈ key rate × window span × per-key/event state, multiplied by overlapping windows where applicable.

Числа выше — учебные inputs или размерностные формулы. Их нельзя выдавать за benchmark или SLA конкретного продукта.

Связанные темы

[CONCEPT]stream-processing

[CONCEPT]streaming-joins

Первичные источники

• https://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf
• https://nightlies.apache.org/flink/flink-docs-stable/docs/concepts/time/
• https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/datastream/operators/windows/
• https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/datastream/event-time/generating_watermarks/

Scope note

Диаграмма показывает причинные границы и recovery contracts, а не скрытую реализацию конкретного managed-сервиса. Любая stronger guarantee действует только в явно названной transaction/checkpoint/acknowledgement boundary.