Change data capture: snapshot handoff, ordered logs and idempotent apply

CDC переносит committed changes, но не создаёт автоматически «exact replica». Нужно определить snapshot/stream boundary, transaction ordering scope, keys, DDL evolution, delete/tombstone semantics, offset durability и log retention.

Корректная модель

• Log-based CDC reads committed changes; snapshot establishes historical state.
• Snapshot-to-stream handoff needs an explicit source position and tolerates identifiable overlap.
• Ordering is source/transaction/partition scoped; consumers must not assume one global total order across tables.
• Replication slots retain source WAL and are a production disk-risk boundary.

Границы и компоненты

Сценарии

Consistent snapshot handoff

The connector records LSN P and scans a consistent snapshot. After snapshot completion it streams from P; overlap can duplicate a row change and must be deduplicated by key/version.

Проверяемый исход: No committed update between snapshot and streaming is missed.

Connector or consumer restart

Offset persistence can lag published/applied work. Restart replays a suffix. The target compares source position/version and keeps one logical state transition.

Проверяемый исход: At-least-once delivery does not become duplicate business rows.

Schema change contract

A DDL change can alter event schemas and old/new values. The schema gate verifies consumer compatibility before production rollout; missing primary keys or replica identity are explicit blockers.

Проверяемый исход: Incompatible DDL pauses/fails the pipeline visibly instead of silently dropping fields or misidentifying rows.

Slot lag and disk pressure

A stopped connector prevents the slot from advancing, so WAL accumulates. The source database can run out of disk if operations only watch broker lag.

Проверяемый исход: Alerts trigger throttling, recovery or deliberate resnapshot/drop decision before disk exhaustion; no silent slot deletion.

Failure, concurrency и replay checklist

• Use stable primary keys or an explicitly documented identity strategy.
• Persist target state and last source version atomically per key.
• Monitor retained WAL bytes/LSN lag, not only connector/broker lag.
• Gate DDL and test deletes, tombstones, large transactions, failover and resnapshot.

Формулы, units и допущения

• Retained WAL bytes ≈ source WAL bytes/s × connector outage seconds, plus bursts and safety margin.
• Catch-up time = backlog bytes / (sustainable decode/apply bytes/s − new WAL bytes/s), requiring a positive denominator.
• Dedup identity should include source partition/table/key and ordered position or transaction event index; timestamp alone is insufficient.

Числа выше — учебные inputs или размерностные формулы. Их нельзя выдавать за benchmark или SLA конкретного продукта.

Связанные темы

[CONCEPT]database-migration-strategies

[CONCEPT]exactly-once-semantics

Первичные источники

• https://debezium.io/documentation/reference/stable/connectors/postgresql.html
• https://www.postgresql.org/docs/current/logicaldecoding.html
• https://www.postgresql.org/docs/current/logical-replication-architecture.html
• https://www.postgresql.org/docs/current/warm-standby.html#STREAMING-REPLICATION-SLOTS

Scope note

Диаграмма показывает причинные границы и recovery contracts, а не скрытую реализацию конкретного managed-сервиса. Любая stronger guarantee действует только в явно названной transaction/checkpoint/acknowledgement boundary.