Exactly-once semantics: name the transaction boundary

Exactly once — не свойство сообщения в вакууме. Это proof, что один logical input создаёт один logical effect в конкретной системе/transaction boundary несмотря на retries and crashes.

Корректная модель

• Kafka EOS is scoped to Kafka transactions/read_committed processing, not arbitrary external databases.
• Flink exactly-once state requires replayable sources; end-to-end requires transactional/idempotent sinks.
• Timeout after a side effect is UNKNOWN, not evidence of failure.
• Dedup state must be atomic and retained for the full replay/retry horizon.

Границы и компоненты

Сценарии

Kafka-scoped read-process-write

A transactional producer writes output records and the consumed offsets in one Kafka transaction. read_committed consumers hide aborted outputs.

Проверяемый исход: Exactly-once is claimed only for the Kafka input/output transaction and correctly handled rebalances/errors.

Local transaction and outbox

A domain change and outbox intent commit in one database transaction. Relay delivery may duplicate, so the consumer remains idempotent.

Проверяемый исход: No dual-write gap exists between local domain state and publication intent.

External ambiguous outcome

Intent is durable before calling the provider. A timeout can mean success; state becomes UNKNOWN and no blind second charge is issued.

Проверяемый исход: Reconciliation queries by the same provider idempotency key and finalizes one stable result.

Dedup retention and replay horizon

A replay older than dedup state can create a second logical effect. TTL must exceed the maximum source replay, client retry and disaster recovery horizon or an authoritative permanent key must exist.

Проверяемый исход: Expired state fails closed or reconciles against durable domain identity; invalid Redis command folklore is removed.

Failure, concurrency и replay checklist

• Persist intent/fingerprint before network side effects.
• Never use check then external call then mark as an exactly-once protocol.
• Keep reconciliation leases/fencing and an auditable state machine.
• Do not put unguarded side effects inside retried database/stream closures.

Формулы, units и допущения

• Dedup retention >= max(client retry horizon, broker replay retention used by recovery, backup restore replay horizon) plus clock/operational margin.
• If unknown outcomes arrive at u/s and reconciliation clears r/s, backlog is stable only when r > u.
• Exactly-once proof is logical-set cardinality by operation ID, not physical invocation count.

Числа выше — учебные inputs или размерностные формулы. Их нельзя выдавать за benchmark или SLA конкретного продукта.

Связанные темы

[CONCEPT]idempotency

[CONCEPT]apache-flink-deep

Первичные источники

• https://kafka.apache.org/41/design/design/
• https://kafka.apache.org/41/javadoc/org/apache/kafka/clients/producer/KafkaProducer.html
• https://nightlies.apache.org/flink/flink-docs-stable/docs/learn-flink/fault_tolerance/
• https://www.postgresql.org/docs/current/sql-insert.html

Scope note

Диаграмма показывает причинные границы и recovery contracts, а не скрытую реализацию конкретного managed-сервиса. Любая stronger guarantee действует только в явно названной transaction/checkpoint/acknowledgement boundary.