Golden signals: distributions, demand, failures, and saturation

Google’s four golden signals are latency, traffic, errors, and saturation. They form a compact service-monitoring checklist, not a proof that a system is healthy whenever four widgets are green. Each signal needs a workload unit, population, dimensions, window, data-quality policy, and a user-impact interpretation.

Latency is a distribution and successful and failed requests may need separate views. Traffic describes demand in the system’s own units. Errors include explicit failures, policy failures, and incorrect or too-slow responses. Saturation is proximity to a limiting resource and can appear as queueing or constrained capacity before nominal utilization reaches 100%.

Модель и предпосылки

• Classic Prometheus histograms estimate p99 with histogram_quantile(0.99, sum by (le, route) (rate(http_request_duration_seconds_bucket[5m]))); the estimate depends on bucket placement and interpolation.
• A percentile cannot be reconstructed from a mean or a few example values. An exact teaching distribution must state its entire relevant count/bucket population.
• Adding API replicas while a database or connection pool is saturated can increase downstream demand and worsen an outage; scaling follows a tested bottleneck model and safety limit.

Проверяемые утверждения

• C1. The four golden signals are latency, traffic, errors, and saturation; successful and failed request latency should be distinguished when failure is faster or slower.
• C2. Prometheus histogram quantiles require bucket rates and le in the aggregation for classic histograms; the result is an estimate, not an exact order statistic.
• C3. Percentile values are not implied by the mean or by a verbal “small slow tail”; relevant event counts or histogram buckets must be given.
• C4. Saturation and overload can cascade through queues, retries, dependencies, and resource exhaustion.
• C5. Pages should represent actionable user impact or imminent failure; dashboards may retain richer diagnostic signals.

Исполняемые сценарии

Latency distribution with valid PromQL. Histogram buckets describe the measured population; p99 is estimated from correctly aggregated bucket rates rather than inferred from an average.

Traffic and errors retain context. Demand and failure rates use explicit units and eligible populations instead of a single global request count.

Saturation before visible failure. Connection-pool waiters and database pressure appear before the service exhausts capacity.

Safe response to downstream saturation. The controller limits demand and tests the bottleneck rather than blindly adding callers to an overloaded database.

Ошибки проектирования

• Не выводите p99 из average или пары точек; задайте population/buckets и window.
• Не считайте HTTP 200 успешным, если user journey нарушает correctness или latency objective.
• Не называйте CPU единственной saturation-метрикой; измеряйте очереди, pool waits, memory pressure и downstream limits.
• Не масштабируйте caller вслепую, когда ограничение находится в database или queue consumer.

Границы гарантии

• Golden signals описывают service symptoms; dependency metrics, traces, profiles and logs нужны для диагностики.
• Aggregation can hide tenant, region, route or replica failures; dimensions require bounded cardinality.
• Alert thresholds and scaling policies are workload-specific and need load/failure tests.

Связанные материалы

Официальные и первичные источники

1. Google SRE Book: Monitoring Distributed Systems
2. Prometheus: Histograms and summaries
3. Google SRE Book: Addressing Cascading Failures
4. Google SRE Book: Production Services Best Practices