System Design Cases
Apache Iceberg / Delta Lake / Hudi — open table formats over object storage
Open table formats (Apache Iceberg, Delta Lake, Apache Hudi) — metadata layer over Parquet/ORC files on S3/GCS/ADLS that adds ACID transactions, snapshot isolation, time-travel, schema/partition evolution, efficient updates. Lakehouse architecture: cheap object storage + warehouse semantics. Catalogs: REST, Polaris, Nessie, Glue, Hive Metastore, Unity Catalog. Scenarios: atomic write commit (parquet + manifest + CAS), concurrent writers with optimistic retry, time-travel query (timestamp → snapshot id), MoR vs CoW trade-off with compaction, schema/partition evolution without rewrite, full read path with manifest pruning, maintenance (compaction, snapshot expiration, orphan cleanup). Includes ADRs on Iceberg vs Delta vs Hudi selection and CoW vs MoR write-vs-read trade-off.
Iceberg, Delta Lake and Hudi: transactional table formats on object storage
Table format управляет snapshots, schema/partition evolution, concurrency and file lifecycle. Он не является object store, query engine или magical ACID layer for arbitrary external systems.
Корректная модель
- All three are table formats/protocols over files and require compatible catalogs/engines.
- Iceberg snapshots/manifests, Delta log versions/checkpoints and Hudi timeline/file groups are different mechanics.
- Optimistic concurrency requires conflict validation and retries; object creation alone is not a table commit.
- Compaction, orphan cleanup and history expiration are correctness-sensitive maintenance.
Границы и компоненты
| Компонент | Ответственность |
|---|---|
| Writer A | Создаёт immutable data/delete/log files from one base snapshot. |
| Concurrent Writer B | Может конфликтовать и обязан validate/retry by protocol. |
| Catalog or Commit Coordinator | Atomically publishes current table metadata/version under supported contract. |
| Object Storage Files | Хранит immutable data and metadata objects; не определяет table semantics alone. |
| Iceberg Snapshots and Manifests | Tracks file sets, metrics, schemas and partition specs. |
| Delta Transaction Log | Ordered actions and checkpoints reconstruct MVCC table versions. |
| Hudi Timeline and File Groups | Tracks instants plus CoW/MoR base/log file lifecycle. |
| Compatible Query Engines | Read a pinned snapshot/version using implemented protocol features. |
| Compaction and Expiration | Rewrites small/log files and safely removes unreachable history after gates. |
Сценарии
Iceberg snapshot and manifest commit
Writer A creates data files and metadata/manifests from base snapshot S, then atomically swaps the current metadata pointer through a catalog/filesystem contract. Readers pinned to S stay consistent.
Проверяемый исход: Uncommitted/orphan files are not visible table state and are cleaned only by safe maintenance.
Delta concurrent commit and retry
Two writers read version v and attempt a write. Delta protocol/implementation checks conflicts and serializes log versions; a conflicting transaction re-reads and retries rather than overwriting v+1.
Проверяемый исход: Concurrent success is allowed only when the protocol says operations do not conflict.
Hudi Copy-on-Write versus Merge-on-Read
CoW rewrites affected base-file groups for updates; MoR appends changes to log/base files and later compacts, shifting cost between write and read.
Проверяемый исход: Choice is based on measured update/read/compaction needs, not a claim one mode is always faster.
Safe snapshot expiration and file cleanup
Old snapshots/log versions can serve readers, rollback, audit or streaming jobs. Maintenance proves they are outside retention and no supported reader references them before deleting files.
Проверяемый исход: VACUUM/expiration never runs from an invented rewrite speed or merely because a new commit exists.
Failure, concurrency и replay checklist
- Pin exact protocol/table feature versions supported by every engine.
- Treat commit timeout as ambiguous and resolve current metadata/version before retry.
- Track orphan files, small files, metadata growth and failed maintenance.
- Retain history longer than maximum reader/stream/rollback/backup requirement and test restore.
Формулы, units и допущения
- File rewrite time = bytes rewritten / sustained end-to-end throughput; never use an unmeasured 500 GB/s claim.
- Small-file count drives planning/metadata overhead separately from total bytes.
- Update amplification depends on file-group overlap, delete representation and compaction policy; measure per workload.
Числа выше — учебные inputs или размерностные формулы. Их нельзя выдавать за benchmark или SLA конкретного продукта.
Связанные темы
Первичные источники
- https://github.com/apache/iceberg/blob/main/format/spec.md
- https://github.com/delta-io/delta/blob/master/PROTOCOL.md
- https://docs.delta.io/concurrency-control/
- https://hudi.apache.org/learn/tech-specs/
- https://hudi.apache.org/docs/table_types/
Scope note
Диаграмма показывает причинные границы и recovery contracts, а не скрытую реализацию конкретного managed-сервиса. Любая stronger guarantee действует только в явно названной transaction/checkpoint/acknowledgement boundary.