Caching patterns and their failure semantics

A cache is a derived-data dependency between the application and its source of truth. The application owns key construction, freshness, invalidation and fallback policy. Clients should not bypass that policy by reading the cache or database directly.

Cache-aside

On a read, the application checks the cache, loads the authoritative source after a miss, and fills the cache. A correct key contains every input that can change the response: object id, tenant or ACL scope, locale, filters, representation and relevant policy/model versions.

TTL limits how long an entry may survive without another signal; it does not guarantee freshness. Jitter spreads expirations. Negative caching can protect the source for genuine “not found” results, but authorization failures and transient errors must not be cached as shared absence.

A late fill can overwrite a newer value. Store a version with the value, use compare-and-set where available, or invalidate after the fill window. Do not assume that one delete eliminates every race.

Writes and invalidation

The database commit is authoritative. After commit, invalidate or update affected keys. If losing the invalidation would violate the freshness objective, record it in the same transaction as an outbox event and process it idempotently. “Write-through” is not a promise that cache and database can never diverge: partial failure still needs ordering and recovery.

Write-behind acknowledges before authoritative storage is committed. A volatile cache alone cannot make that durable. A safe design needs a durable ordered log, replay, idempotent writes, poison-record handling, bounded lag and an explicit read-your-writes story.

Stampedes

When a hot key expires, request coalescing or single-flight lets one owner fill while peers wait. Recheck the cache after ownership is acquired. Bound ownership and downstream deadlines. A distributed lock additionally needs unique ownership tokens, safe release and protection against expired owners; a lease without fencing does not make arbitrary side effects safe.

Stale-while-revalidate is acceptable only when the product defines how stale is safe. Security, revocation and money-related data often require a fail-closed or much tighter policy.

Cache failure and cache addiction

A cache outage can multiply database traffic. Fallback is safe only if the source has tested headroom. Use short cache timeouts, admission control, per-key coalescing, concurrency limits, load shedding and graceful degradation. Monitor source load during cache-hit degradation; “the application still works without cache” is an operational claim that must be load-tested.

Scenarios

▶Walkthrough: cache-aside-read (not found)

Versioned cache-aside read and guarded fill.

▶Walkthrough: write-invalidate (not found)

Authoritative commit followed by recoverable invalidation.

▶Walkthrough: stampede (not found)

Single-flight coalescing for a burst on one hot key.

▶Walkthrough: cache-failure (not found)

Cache timeout with bounded fallback rather than an uncontrolled source storm.

▶Walkthrough: write-behind (not found)

The durability boundary that a real write-behind design must cross.

Measure

Track hit ratio by key class, miss amplification, fill concurrency, eviction reasons, stale-serving age, invalidation lag, source QPS during cache incidents, and p50/p95/p99 separately for hits, misses and timeouts. Size the cache from working-set measurements and eviction behavior, not a universal percentage.

Primary sources

• Amazon Builders' Library, Caching challenges and strategies: https://aws.amazon.com/builders-library/caching-challenges-and-strategies/
• Nishtala et al., Scaling Memcache at Facebook: https://www.usenix.org/system/files/conference/nsdi13/nsdi13-final170_update.pdf
• DeCandia et al., Dynamo: Amazon's Highly Available Key-value Store: https://web.stanford.edu/class/cs244/papers/amazon-dynamo-sosp2007.pdf