Memory hierarchy: boundaries, math and failure modes

A memory hierarchy is not one universal ladder of fixed latency numbers. It is a set of hardware, kernel and distributed-service boundaries with different units, ownership and failure modes. CPU caches and DRAM are load/store paths. The page cache is kernel-managed file data in DRAM. A block device is explicit I/O. Object storage is a remote API, not a transparent “next cache level”.

Latency math that stays dimensionally correct

For a clock frequency f_GHz, one cycle lasts 1 / f_GHz nanoseconds. A measured latency of L_ns therefore costs approximately:

cycles = L_ns * f_GHz

At 3 GHz, one cycle is about 0.333 ns and 100 ns is about 300 cycles. Frequency can vary with power state and architecture, so this identity does not turn a benchmark from another machine into a guarantee. Measure the target CPU, access pattern, NUMA placement and concurrency. Report a distribution, not one memorized integer.

CPU caches, address translation and DRAM

The core first needs a valid virtual-to-physical translation and permissions. A TLB miss is not automatically a page fault: hardware may complete a page-table walk and cache the translation. A page fault is an exception. It may be expected, such as lazy allocation or copy-on-write, or fatal, such as an invalid access.

A cache miss moves a cache-line request toward lower cache levels and the memory controller. DRAM latency depends on locality, contention, row state, frequency and platform. Bandwidth and latency are different: many outstanding accesses can sustain bandwidth while an individual dependency chain still waits.

Page cache and page faults

The kernel page cache stores file-backed pages in memory. A minor fault can often be resolved without device I/O, for example by installing a mapping to a resident page. A major fault requires I/O and can block the faulting thread. CPU iowait accounting is not “the CPU doing the disk read”; it is scheduler/accounting context while work waits for I/O.

Writes usually dirty cached pages before writeback. write() success, page-cache visibility and durable media persistence are separate contracts. Slow writeback and reclaim can produce latency cliffs later. Watch dirty bytes, writeback time, major faults, pressure-stall information, device queue depth and tail latency.

Storage engines are workload choices

An LSM tree buffers and sorts writes, then compacts immutable runs. It can turn random updates into more sequential I/O, including on rotating media. It does not “die on HDD” by definition. Read amplification, write amplification, compaction bandwidth, cache hit rate and the workload’s latency objective decide whether a particular design is appropriate.

Object storage is a service

An object request crosses client code, DNS/connection management, TLS, network queues, authorization and provider storage. Retries must be bounded and operation-aware. Do not compare an object-service p99 with an L1 hit as though both were interchangeable memory accesses. Cache objects locally only with an explicit freshness, eviction, integrity and authorization policy.

Scenarios

A core-local cache hit, with latency treated as a measurement.

A last-level-cache miss traverses the memory controller and DRAM, then refills upper caches.

A valid file mapping is absent from the page cache, so the thread blocks for device I/O and resumes after the mapping is installed.

Dirty-page writeback exposes queueing and reclaim pressure instead of pretending writes are instantly durable.

An object read is modeled as a remote authenticated service request with independent tails and retries.

Operational checklist

• Benchmark representative working sets, access strides, read/write mixes and concurrency on the production hardware.
• Separate TLB misses, cache misses, minor faults, major faults and explicit I/O in telemetry.
• Preserve NUMA locality or measure the penalty when threads and memory move.
• Put deadlines and bounded retries around remote storage; record attempt count and final outcome.
• Define the exact durability point for every write path.

Primary sources

• Intel 64 and IA-32 Architectures Optimization Reference Manual: https://www.intel.com/content/www/us/en/developer/articles/technical/intel-sdm.html
• Linux page tables, TLBs and faults: https://www.kernel.org/doc/html/latest/mm/page_tables.html
• Linux memory-management documentation: https://www.kernel.org/doc/html/latest/admin-guide/mm/index.html
• O'Neil et al., The Log-Structured Merge-Tree: https://www.cs.umb.edu/~poneil/lsmtree.pdf
• Amazon S3 API reference: https://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html