Iceberg, Delta Lake and Hudi: transactional table formats on object storage

Table format управляет snapshots, schema/partition evolution, concurrency and file lifecycle. Он не является object store, query engine или magical ACID layer for arbitrary external systems.

Корректная модель

• All three are table formats/protocols over files and require compatible catalogs/engines.
• Iceberg snapshots/manifests, Delta log versions/checkpoints and Hudi timeline/file groups are different mechanics.
• Optimistic concurrency requires conflict validation and retries; object creation alone is not a table commit.
• Compaction, orphan cleanup and history expiration are correctness-sensitive maintenance.

Границы и компоненты

Сценарии

Iceberg snapshot and manifest commit

Writer A creates data files and metadata/manifests from base snapshot S, then atomically swaps the current metadata pointer through a catalog/filesystem contract. Readers pinned to S stay consistent.

Проверяемый исход: Uncommitted/orphan files are not visible table state and are cleaned only by safe maintenance.

Delta concurrent commit and retry

Two writers read version v and attempt a write. Delta protocol/implementation checks conflicts and serializes log versions; a conflicting transaction re-reads and retries rather than overwriting v+1.

Проверяемый исход: Concurrent success is allowed only when the protocol says operations do not conflict.

Hudi Copy-on-Write versus Merge-on-Read

CoW rewrites affected base-file groups for updates; MoR appends changes to log/base files and later compacts, shifting cost between write and read.

Проверяемый исход: Choice is based on measured update/read/compaction needs, not a claim one mode is always faster.

Safe snapshot expiration and file cleanup

Old snapshots/log versions can serve readers, rollback, audit or streaming jobs. Maintenance proves they are outside retention and no supported reader references them before deleting files.

Проверяемый исход: VACUUM/expiration never runs from an invented rewrite speed or merely because a new commit exists.

Failure, concurrency и replay checklist

• Pin exact protocol/table feature versions supported by every engine.
• Treat commit timeout as ambiguous and resolve current metadata/version before retry.
• Track orphan files, small files, metadata growth and failed maintenance.
• Retain history longer than maximum reader/stream/rollback/backup requirement and test restore.

Формулы, units и допущения

• File rewrite time = bytes rewritten / sustained end-to-end throughput; never use an unmeasured 500 GB/s claim.
• Small-file count drives planning/metadata overhead separately from total bytes.
• Update amplification depends on file-group overlap, delete representation and compaction policy; measure per workload.

Числа выше — учебные inputs или размерностные формулы. Их нельзя выдавать за benchmark или SLA конкретного продукта.

Связанные темы

[CONCEPT]etl-vs-elt

[CONCEPT]data-mesh

Первичные источники

• https://github.com/apache/iceberg/blob/main/format/spec.md
• https://github.com/delta-io/delta/blob/master/PROTOCOL.md
• https://docs.delta.io/concurrency-control/
• https://hudi.apache.org/learn/tech-specs/
• https://hudi.apache.org/docs/table_types/

Scope note

Диаграмма показывает причинные границы и recovery contracts, а не скрытую реализацию конкретного managed-сервиса. Любая stronger guarantee действует только в явно названной transaction/checkpoint/acknowledgement boundary.