RED and USE: complementary views, not causal guarantees

RED asks for request rate, errors, and duration for a service. USE asks for utilization, saturation, and errors for each resource. They are complementary checklists: RED bounds user-visible symptoms while USE helps enumerate resource bottlenecks. Neither framework proves a root cause or guarantees coverage of every failure mode.

A percentile threshold is a latency condition, not an SLO burn-rate calculation. Burn requires a defined bad-event ratio and allowed ratio. Histograms estimate distributions with bucket or schema trade-offs; native histograms improve some operational choices but do not provide universal exactness, storage reduction, or availability in every component.

Модель и предпосылки

• Requests traverse user → load balancer → service → datastore. The diagram never skips the load balancer for convenience.
• RED labels are bounded by service/route/status class. USE definitions are resource-specific: CPU queue, memory reclaim, disk latency/queue, and pool waiters are not interchangeable.
• Average utilization can hide short saturation bursts; high utilization without a queue is not automatically an incident.

Проверяемые утверждения

• C1. RED is rate, errors, and duration for request-driven services; USE is utilization, saturation, and errors applied to each resource.
• C2. USE is an early systematic checklist, not a complete causal model; definitions and data quality depend on the resource.
• C3. A classic Prometheus p99 estimate uses histogram_quantile(0.99, sum by (le) (rate(..._bucket[5m]))); accuracy depends on bucket boundaries and the observed distribution.
• C4. Native histograms use sparse schemas and can merge resolution under limits, but deployment, interpolation error, and storage cost remain configuration-dependent.
• C5. A p99 threshold is not a burn alert until latency-good events, an SLO allowance, and a window turn it into a bad-event ratio.

Исполняемые сценарии

RED on the real service path. The request is measured on the same path the user takes, including the load balancer.

USE saturation drill-down. A latency symptom is correlated with a resource queue, then tested before it is called causal.

Histogram quantile boundary. The team aggregates classic buckets correctly and records interpolation limits rather than promising exact p99.

Latency SLO versus bare p99. A latency objective converts eligible slow events into a bad-event ratio before burn is calculated.

Ошибки проектирования

• Не называйте p99 > 500 ms “14x burn” без SLO и доли bad events.
• Не считайте 50% CPU безопасным при длинной run queue и не считайте 90% CPU неисправностью без saturation/error evidence.
• Не агрегируйте summary quantiles между экземплярами как обычные числа.
• Не обещайте native histograms точнее и дешевле при любом workload и backend.

Границы гарантии

• RED лучше соответствует request-driven сервисам; batch, stream и offline workloads требуют своих единиц работы и freshness/correctness indicators.
• USE показывает место давления, но причинность проверяется контролируемым изменением, профилем или трассой.
• Метрика с неверной семантикой или missing data не становится истинной от красивого dashboard.

Связанные материалы

Официальные и первичные источники

1. Grafana: The RED Method
2. Brendan Gregg: The USE Method
3. Prometheus: Histograms and summaries
4. Prometheus: Native histogram specification
5. Google SRE Workbook: Alerting on SLOs