Performance engineering: measurement, profiling, database maintenance, and safe tail mitigation

Performance engineering is an evidence loop: define a user/workload objective, measure a representative baseline, localize the limiting resource or path, form a hypothesis, change one controlled factor, and compare against correctness and resource cost. Metrics, traces and profiles answer different questions and each can be biased or intrusive.

Percentiles expose distribution tails but are not universally superior to every other statistic. Means, medians, percentiles, histograms, throughput, concurrency, queueing and resource utilization all have valid roles when tied to a decision. Synthetic benchmarks require workload fidelity and production measurements require safety, privacy and overhead controls.

Модель и предпосылки

• Ordinary PostgreSQL VACUUM uses a lock compatible with normal reads/writes and can cause I/O pressure; VACUUM FULL takes ACCESS EXCLUSIVE. Tail truncation can also briefly seek a stronger lock. “Autovacuum locks the table for two seconds” is not a valid general claim.
• Hedged requests send a delayed duplicate and cancel the loser to trade bounded extra work for lower tail latency. They require safely repeatable/idempotent semantics, replica capacity, a delay/load budget and cancellation; duplicate writes are unsafe without stronger deduplication semantics.
• Profiler overhead and bias are tool, language, sampling-rate and workload dependent. Measure overhead in a canary and do not publish a universal one-percent number.

Проверяемые утверждения

• C1. Tail latency can dominate fan-out services; delayed hedging and cancellation can reduce tails with bounded extra load in the measured workload.
• C2. Plain PostgreSQL VACUUM takes SHARE UPDATE EXCLUSIVE and normally runs with reads/writes; VACUUM FULL takes ACCESS EXCLUSIVE, while vacuum I/O and truncation can still affect latency.
• C3. Profiles identify expensive code paths but profilers can perturb execution and sampling can bias what is visible.
• C4. Automatic retries and hedges are safe only for operations with repeatable semantics or an application-level deduplication contract.
• C5. Performance results include correctness, throughput, latency distribution and resource cost under a stated workload; a single before/after p99 does not prove general improvement.

Исполняемые сценарии

Representative baseline. The engineer fixes workload, correctness, windows and resource metrics before interpreting a latency distribution.

Profile and validate a hypothesis. A sampling profile localizes CPU work, but a canary experiment verifies latency and resource changes under the same workload.

Plain VACUUM lock semantics. Routine vacuum coexists with normal DML but its I/O and exceptional stronger-lock phases remain observable.

Bounded idempotent hedge. A delayed duplicate is used only for a safe read, with a load budget and cancellation of the losing request.

Ошибки проектирования

• Не оптимизируйте synthetic microbenchmark, не проверив production workload shape and correctness.
• Не обещайте fixed profiler overhead или точный flame graph при неизвестном sampling bias.
• Не используйте hedged writes/payment calls без idempotency/deduplication and load control.
• Не путайте plain VACUUM, VACUUM FULL, autovacuum I/O и краткий truncation lock.

Границы гарантии

• A faster median can hide a worse tail or error rate; every comparison uses the same eligible population and confidence method.
• Tail techniques can amplify load during saturation; adaptive budgets must disable them before a positive feedback loop.
• Profiling endpoints require authentication, least privilege and sensitive-data review.

Связанные материалы

Официальные и первичные источники

1. Google Research: The Tail at Scale
2. The Tail at Scale paper
3. PostgreSQL: Explicit Locking
4. PostgreSQL: Routine Vacuuming
5. Go Diagnostics and pprof
6. RFC 9110: HTTP Semantics and Idempotent Methods