System Design Cases
Apache Flink Deep Dive
Apache Flink deep dive: true streaming engine with stateful operators, RocksDB state backend, Chandy-Lamport distributed snapshots for exactly-once via 2PC sinks. Shows JobManager + 3 TaskManagers (source, window aggregator, sink), RocksDB local state, S3 snapshot storage, Kafka transactional sink + Postgres XA sink. Five scenarios: stateful keyed aggregation, checkpoint barrier propagation, 2PC sink commit, failure recovery with tx abort, savepoint-based version upgrade. Includes 3 ADRs comparing Flink vs Spark Structured Streaming vs Kafka Streams.
Apache Flink: state, checkpoint barriers and end-to-end guarantees
Flink выполняет stateful stream graph и восстанавливается из consistent snapshots. Главная граница: exactly-once state effects внутри job не превращает Kafka, PostgreSQL и внешний API в одну transaction.
Корректная модель
- Checkpoint contains source positions and operator state before numbered barriers.
- Exactly-once means each logical event affects managed state once after recovery, not one physical execution.
- End-to-end exactly once needs replayable sources and transactional or idempotent sinks.
- Savepoints serve operational upgrades; checkpoint interval and result visibility are separate.
Границы и компоненты
| Компонент | Ответственность |
|---|---|
| Replayable Source | Предоставляет partition offsets и replay after restore. |
| Checkpoint Coordinator | Запускает numbered barriers и принимает successful acknowledgements. |
| Stateful Operator A | Обрабатывает keyed events и snapshots local/operator state. |
| Stateful Operator B | Aligns multiple inputs or captures in-flight buffers for unaligned mode. |
| Durable Checkpoint Store | Хранит recoverable snapshot metadata/files. |
| Transactional Kafka Sink | Может commit output with checkpoint through its connector contract. |
| Idempotent Database Sink | Применяет stable event IDs/versions; не атомарен с Kafka sink автоматически. |
| Job Recovery and Savepoints | Restarts from checkpoint or controlled savepoint with stable operator IDs. |
Сценарии
Aligned checkpoint barriers
Barriers divide pre/post-checkpoint events. A multi-input operator pauses already-barriered channels until the others catch up, then snapshots state.
Проверяемый исход: Checkpoint completion is a recovery point, not a claim that business output waited one checkpoint interval to become visible.
Failure and replay
After a task failure, Flink restores the latest completed checkpoint and rewinds replayable sources. Some records execute again, but managed state reflects each logical input once under exactly-once mode.
Проверяемый исход: Duplicate execution is expected; sink contract determines whether duplicate external effects appear.
Sink guarantee boundaries
Kafka output can participate via connector transactions; a database sink may be idempotent or transactional under a separate contract. Writing both does not create one cross-system atomic commit.
Проверяемый исход: Every sink reports its own checkpoint/idempotency proof; reconciliation handles partial cross-sink success.
Backpressure and controlled upgrade
Slow downstream work delays aligned barriers. Unaligned checkpoints can capture in-flight data and shorten checkpoint completion, but do not remove processing latency. Savepoints with stable operator IDs support controlled upgrades.
Проверяемый исход: Operations distinguish checkpoint health, record latency and savepoint compatibility.
Failure, concurrency и replay checklist
- Stable operator IDs and tested state schema compatibility for upgrades.
- Checkpoint storage must outlive failed workers and be independently durable.
- Monitor alignment duration, checkpoint size/failures, backpressure and restart loops.
- Reconcile multiple sinks; never call them one atomic transaction without a real protocol.
Формулы, units и допущения
- Checkpoint interval bounds recovery progress only under successful completion; replay work also depends on failure time and source rate.
- Backlog growth = input records/s − sustainable sink records/s; unaligned checkpoints do not change that sign.
- State size for a keyed window depends on active keys × retained events/aggregates × serialization overhead, not a fixed RocksDB ratio.
Числа выше — учебные inputs или размерностные формулы. Их нельзя выдавать за benchmark или SLA конкретного продукта.
Связанные темы
[CONCEPT]exactly-once-semantics
Первичные источники
- https://nightlies.apache.org/flink/flink-docs-stable/docs/learn-flink/fault_tolerance/
- https://nightlies.apache.org/flink/flink-docs-stable/docs/ops/state/large_state_tuning/
- https://nightlies.apache.org/flink/flink-docs-stable/docs/ops/state/savepoints/
- https://nightlies.apache.org/flink/flink-docs-stable/docs/connectors/datastream/guarantees/
Scope note
Диаграмма показывает причинные границы и recovery contracts, а не скрытую реализацию конкретного managed-сервиса. Любая stronger guarantee действует только в явно названной transaction/checkpoint/acknowledgement boundary.