S3-style object storage: immutable versions, checksums, and repair

The diagram is a pedagogical S3-style design, not a claim about undisclosed AWS internals. The six storage nodes illustrate one representative 4+2 erasure-coded stripe so the arithmetic and failure surface are visible. AWS can change internal placement and coding without changing its public contract; rely on the service API and official quotas, not this illustrative stripe.

API and consistency contract

Current Amazon S3 documents strong read-after-write consistency for successful object PUT/DELETE operations and for subsequent GET and LIST operations. A single object-key update is atomic: readers do not receive a partially committed body. Cross-region replication is a separate asynchronous process, so a replica region can legitimately lack the new version after the source-region commit. Multi-Region Access Points route requests; they do not make missing replicas appear synchronously.

Metadata stores immutable version records. A version becomes visible only after the required fragments and explicit checksum have crossed their durability boundary. Versioning turns overwrite and delete into new metadata states; a delete marker is not physical erasure of every older version. Retention, legal hold, and lifecycle rules must be evaluated against versions, not only the latest key name.

ETag is an opaque validator. It is not universally an MD5 digest, especially for multipart uploads and encrypted objects. Clients that need integrity provide and verify a supported checksum algorithm. Conditional writes use If-None-Match or If-Match to prevent create/overwrite races; an ETag mismatch fails without applying the new version.

Multipart and coding math

The worked example uploads 4 TiB using 512 MiB parts:

• 4 TiB = 4 * 1024 * 1024 MiB = 4,194,304 MiB.
• 4,194,304 / 512 = 8,192 parts, below the documented 10,000-part limit.
• Parts can arrive independently and a repeated part number replaces that part for the upload. Completion supplies the ordered part list; abandoned sessions and fragments need explicit cleanup.

For the illustrative 4+2 stripe, four equal data fragments produce two parity fragments. Physical storage is 6 / 4 = 1.5x logical size, or 50% coding overhead before metadata, replicas, padding, and repair headroom. Any four healthy fragments reconstruct the stripe, so two independent fragment failures are tolerable. That fact does not mathematically produce “eleven nines” durability. Correlated rack/zone faults, latent corruption, placement mistakes, detection time, repair bandwidth, and operator error determine the real durability envelope.

No speculative per-TB internal cost or invented annual saving is presented. Cost models use the actual provider price sheet, request mix, storage class, minimum-duration charges, retrieval, replication, and egress for the target region and date.

Request and failure paths

• All SDK traffic follows edge/global routing, regional load balancing, authentication, and policy. There is no direct client-to-storage or load-balancer bypass.
• Public immutable objects may be cached. Private or signed content requires an explicit cache-key and authorization policy; query signatures must not create unsafe sharing or unbounded variants.
• A hot object is not “split across two CPUs” by range sharding. Replicated read serving, metadata caching, CDN delivery, request coalescing, and admission control absorb read skew.
• Temporary 503 Slow Down is retried with bounded exponential backoff, jitter, a deadline, and observability. A timeout on a mutation is ambiguous; multipart IDs, part numbers, version conditions, and idempotency policy prevent accidental duplicate effects.
• Cross-region replication is observed by backlog and replication status. A client that requires read-your-write across regions either pins reads to the source/version until replication completes or uses an application protocol that acknowledges the weaker boundary.

Tiering and lifecycle

Lifecycle policies can transition objects by age and remove expired versions according to policy. Intelligent-Tiering separately monitors access and can move eligible objects after service-defined inactivity windows. Archive implementation details are deliberately abstract: restore latency and retrieval behavior depend on the chosen storage class and retrieval tier, not an assumption about “tape robots.”

Lifecycle workers must be idempotent and race-safe with new versions, legal hold, restores, and deletes. Restore creates or exposes a temporary accessible copy according to the service contract; it does not mutate the archived bytes in place.

Integrity and operations

Scrubbing compares stored checksums, reconstructs corrupt or missing fragments from a sufficient healthy subset, verifies the replacement, then records restored redundancy. Repair is prioritized by remaining fault tolerance and failure-domain risk. Capacity reserves cover rebuild traffic; a full cluster with no repair headroom is not durable merely because erasure coding exists.

Monitor commit latency, incomplete multipart age, checksum failures, orphan bytes, metadata/data divergence, repair queue age, degraded-stripe count, per-failure-domain saturation, 4xx/5xx by operation, 503 retry pressure, hot keys, lifecycle backlog, replication lag, and restore SLA.

Related material

Capacity planning is an explore diagram and is linked as a viewer page rather than a course directive.

Primary sources

• Amazon S3 User Guide — current regional consistency and object-operation contract.
• Amazon S3 multipart upload overview — upload lifecycle, part replacement, ETag caveats, and checksum handling.
• Amazon S3 multipart upload limits — current part-count and part-size quotas; quotas should be rechecked when implementing.
• Amazon S3 checking object integrity — supported explicit checksums and verification.
• Amazon S3 conditional writes — If-None-Match and If-Match concurrency control.
• Amazon S3 replication — asynchronous same- and cross-region replication.
• Amazon S3 Multi-Region Access Point replication — routing can reach a bucket before the object replica exists.
• Amazon S3 performance design patterns — retry/backoff, temporary 503 Slow Down, caching, and hot-object guidance.
• Amazon S3 Intelligent-Tiering — access monitoring, tier transitions, and archive options.
• Amazon S3 lifecycle management — lifecycle actions and version-aware behavior.

Interactive scenarios