Apache Flink: state, checkpoint barriers and end-to-end guarantees

Flink выполняет stateful stream graph и восстанавливается из consistent snapshots. Главная граница: exactly-once state effects внутри job не превращает Kafka, PostgreSQL и внешний API в одну transaction.

Корректная модель

• Checkpoint contains source positions and operator state before numbered barriers.
• Exactly-once means each logical event affects managed state once after recovery, not one physical execution.
• End-to-end exactly once needs replayable sources and transactional or idempotent sinks.
• Savepoints serve operational upgrades; checkpoint interval and result visibility are separate.

Границы и компоненты

Сценарии

Aligned checkpoint barriers

Barriers divide pre/post-checkpoint events. A multi-input operator pauses already-barriered channels until the others catch up, then snapshots state.

Проверяемый исход: Checkpoint completion is a recovery point, not a claim that business output waited one checkpoint interval to become visible.

Failure and replay

After a task failure, Flink restores the latest completed checkpoint and rewinds replayable sources. Some records execute again, but managed state reflects each logical input once under exactly-once mode.

Проверяемый исход: Duplicate execution is expected; sink contract determines whether duplicate external effects appear.

Sink guarantee boundaries

Kafka output can participate via connector transactions; a database sink may be idempotent or transactional under a separate contract. Writing both does not create one cross-system atomic commit.

Проверяемый исход: Every sink reports its own checkpoint/idempotency proof; reconciliation handles partial cross-sink success.

Backpressure and controlled upgrade

Slow downstream work delays aligned barriers. Unaligned checkpoints can capture in-flight data and shorten checkpoint completion, but do not remove processing latency. Savepoints with stable operator IDs support controlled upgrades.

Проверяемый исход: Operations distinguish checkpoint health, record latency and savepoint compatibility.

Failure, concurrency и replay checklist

• Stable operator IDs and tested state schema compatibility for upgrades.
• Checkpoint storage must outlive failed workers and be independently durable.
• Monitor alignment duration, checkpoint size/failures, backpressure and restart loops.
• Reconcile multiple sinks; never call them one atomic transaction without a real protocol.

Формулы, units и допущения

• Checkpoint interval bounds recovery progress only under successful completion; replay work also depends on failure time and source rate.
• Backlog growth = input records/s − sustainable sink records/s; unaligned checkpoints do not change that sign.
• State size for a keyed window depends on active keys × retained events/aggregates × serialization overhead, not a fixed RocksDB ratio.

Числа выше — учебные inputs или размерностные формулы. Их нельзя выдавать за benchmark или SLA конкретного продукта.

Связанные темы

[CONCEPT]exactly-once-semantics

[CONCEPT]windowing-watermarks

Первичные источники

• https://nightlies.apache.org/flink/flink-docs-stable/docs/learn-flink/fault_tolerance/
• https://nightlies.apache.org/flink/flink-docs-stable/docs/ops/state/large_state_tuning/
• https://nightlies.apache.org/flink/flink-docs-stable/docs/ops/state/savepoints/
• https://nightlies.apache.org/flink/flink-docs-stable/docs/connectors/datastream/guarantees/

Scope note

Диаграмма показывает причинные границы и recovery contracts, а не скрытую реализацию конкретного managed-сервиса. Любая stronger guarantee действует только в явно названной transaction/checkpoint/acknowledgement boundary.