System Design Cases
Data Mesh
Data Mesh (Zhamak Dehghani 2019) concept page. 4 principles: domain-oriented decentralized ownership, data as product, self-serve infrastructure, federated computational governance. Shows central DWH bottleneck (Gen 1-2) vs Data Mesh (Gen 3) with domain teams (orders, payments, marketing) owning their own data products on top of self-serve platform (S3+Iceberg, DataHub catalog, Spark/Trino, Monte Carlo) governed by federated council (OPA, data contracts). 5 scenarios: monolithic-bottleneck (legacy), mesh-publish (orders team self-serves), mesh-discover (marketing finds + consumes), contract-violation (governance blocks breaking change), quality-slo (SLO alert routing).
Data mesh: domain-owned products with federated computational governance
Data mesh масштабирует ownership и change across domains. Его unit — usable, discoverable, addressable, trustworthy data product, а не копия таблицы без владельца. Shared platform уменьшает cognitive load; federated governance делает cross-domain ecosystem interoperable.
Корректная модель
- Four principles are collectively: domain ownership, data as a product, self-serve platform and federated computational governance.
- A data product includes data, metadata, code/policy, quality/SLO, discoverability and one accountable owner.
- Federated policy standards need runtime data-plane enforcement, identity/credentials, audit and revocation.
- Central platform capabilities and global interoperability prevent decentralization from becoming incompatible silos.
Границы и компоненты
| Компонент | Ответственность |
|---|---|
| Orders Domain Team | Владеет source semantics, transformations, SLO and incidents for Orders product. |
| Orders Data Product | Versioned contract, data, metadata, quality, policy endpoints and lineage. |
| Customer Domain Team | Независимо владеет Customer product and its evolution. |
| Customer Data Product | Another interoperable product; cross-domain join is a consumer responsibility/product. |
| Self-Serve Data Platform | Paved-road provisioning, compute, storage, CI, observability and identity integration. |
| Catalog and Contract Registry | Discoverability, ownership, schemas, versions, SLOs and lineage. |
| Federated Policy Plane | Defines/checks global standards; returns decisions but is not the data plane. |
| Runtime Access Gateway | Authenticates, authorizes, obtains credentials/tokens, enforces and audits access. |
| Cross-Domain Consumer | Selects compatible product versions and states freshness/purpose requirements. |
| Immutable Access and Change Audit | Records runtime decisions, grants, reads and contract/deployment changes. |
Сценарии
Publish a domain data product
Orders team uses platform templates, registers owner/schema/SLO/purpose policy and publishes only after automated and domain acceptance checks.
Проверяемый исход: The product has one accountable on-call/roadmap owner and a runtime endpoint, not only catalog metadata.
Contract evolution with consumer compatibility
A breaking field semantic change creates a new contract version and migration window. Catalog impact analysis identifies consumers; old version remains until declared deprecation gates pass.
Проверяемый исход: CI checks plus runtime/version telemetry prevent silent consumer breakage.
Runtime access enforcement
Consumer authenticates at the gateway. Gateway asks the federated policy plane for a decision, obtains/scopes short-lived data credentials through the platform integration, enforces row/object/query controls and audits the actual read.
Проверяемый исход: An allow decision alone is never described as credentials, enforcement, audit or revocation.
Quality/security incident ownership
Freshness or contract quality fails after publication. Runtime monitor/gateway blocks or marks affected version, domain owner responds, platform supports rollback, and consumers see status.
Проверяемый исход: Ownership is operational and user-visible; a CI-only pass cannot mask runtime failure.
Failure, concurrency и replay checklist
- One named domain owner/on-call per product; shared ownership ambiguity is a defect.
- Version contracts and measure real consumer usage before deprecation.
- Fail closed on expired identity, policy errors or broken classification for sensitive data.
- Exercise runtime access revocation, product rollback, freshness incidents and audit retrieval.
Формулы, units и допущения
- Product SLO must state freshness, completeness/quality, availability and support boundary with units; a single uptime percentage is insufficient.
- Platform success is measured through lead time, reuse, policy coverage and domain cognitive load, not number of tools.
- Cross-domain joins inherit the slowest relevant freshness/quality contract and need explicit semantic ownership.
Числа выше — учебные inputs или размерностные формулы. Их нельзя выдавать за benchmark или SLA конкретного продукта.
Связанные темы
[CONCEPT]iceberg-deltalake-hudi
Первичные источники
- https://martinfowler.com/articles/data-monolith-to-mesh.html
- https://martinfowler.com/articles/data-mesh-principles.html
- https://martinfowler.com/articles/designing-data-products.html
- https://csrc.nist.gov/pubs/sp/800/207/final
Scope note
Диаграмма показывает причинные границы и recovery contracts, а не скрытую реализацию конкретного managed-сервиса. Любая stronger guarantee действует только в явно названной transaction/checkpoint/acknowledgement boundary.