Change data capture: snapshot handoff, ordered logs and idempotent apply

CDC переносит committed changes, но не создаёт автоматически «exact replica». Нужно определить snapshot/stream boundary, transaction ordering scope, keys, DDL evolution, delete/tombstone semantics, offset durability и log retention.

Корректная модель

• Log-based CDC reads committed changes; snapshot establishes historical state.
• Snapshot-to-stream handoff needs an explicit source position and tolerates identifiable overlap.
• Ordering is source/transaction/partition scoped; consumers must not assume one global total order across tables.
• Replication slots retain source WAL and are a production disk-risk boundary.

Границы и компоненты

Компонент	Ответственность
Source Application	Пишет authoritative business transactions only to source DB.
Source PostgreSQL	Коммитит rows и WAL; остаётся source of truth.
Logical Slot and WAL	Удерживает required log position; может заполнить disk при lag.
CDC Connector	Делает snapshot, читает LSN, schemas and transaction metadata.
Partitioned Change Topics	Хранит change events with stable keys and replay offsets.
Idempotent Projection Consumer	Подключается к topics и conditional-applies source versions.
Target Projection	Хранит last applied source position/version per key.
CDC Health and Schema Gate	Следит за lag/disk, DDL compatibility, offsets и reconciliation.

Сценарии

▶Walkthrough: consistent-snapshot-handoff (not found)

Consistent snapshot handoff

The connector records LSN P and scans a consistent snapshot. After snapshot completion it streams from P; overlap can duplicate a row change and must be deduplicated by key/version.

Проверяемый исход: No committed update between snapshot and streaming is missed.

▶Walkthrough: restart-redelivery (not found)

Connector or consumer restart

Offset persistence can lag published/applied work. Restart replays a suffix. The target compares source position/version and keeps one logical state transition.

Проверяемый исход: At-least-once delivery does not become duplicate business rows.

▶Walkthrough: schema-change-contract (not found)

Schema change contract

A DDL change can alter event schemas and old/new values. The schema gate verifies consumer compatibility before production rollout; missing primary keys or replica identity are explicit blockers.

Проверяемый исход: Incompatible DDL pauses/fails the pipeline visibly instead of silently dropping fields or misidentifying rows.

▶Walkthrough: slot-lag-disk-pressure (not found)

Slot lag and disk pressure

A stopped connector prevents the slot from advancing, so WAL accumulates. The source database can run out of disk if operations only watch broker lag.

Проверяемый исход: Alerts trigger throttling, recovery or deliberate resnapshot/drop decision before disk exhaustion; no silent slot deletion.

Failure, concurrency и replay checklist

• Use stable primary keys or an explicitly documented identity strategy.
• Persist target state and last source version atomically per key.
• Monitor retained WAL bytes/LSN lag, not only connector/broker lag.
• Gate DDL and test deletes, tombstones, large transactions, failover and resnapshot.

Формулы, units и допущения

• Retained WAL bytes ≈ source WAL bytes/s × connector outage seconds, plus bursts and safety margin.
• Catch-up time = backlog bytes / (sustainable decode/apply bytes/s − new WAL bytes/s), requiring a positive denominator.
• Dedup identity should include source partition/table/key and ordered position or transaction event index; timestamp alone is insufficient.

Числа выше — учебные inputs или размерностные формулы. Их нельзя выдавать за benchmark или SLA конкретного продукта.

Связанные темы

[CONCEPT]database-migration-strategies

[CONCEPT]exactly-once-semantics

Первичные источники

• https://debezium.io/documentation/reference/stable/connectors/postgresql.html
• https://www.postgresql.org/docs/current/logicaldecoding.html
• https://www.postgresql.org/docs/current/logical-replication-architecture.html
• https://www.postgresql.org/docs/current/warm-standby.html#STREAMING-REPLICATION-SLOTS

Scope note

Диаграмма показывает причинные границы и recovery contracts, а не скрытую реализацию конкретного managed-сервиса. Любая stronger guarantee действует только в явно названной transaction/checkpoint/acknowledgement boundary.