Observability signals: telemetry, correlation, and evidence limits

Observability is the ability to reason about a system from its outputs. Traces, metrics, and logs are useful telemetry signals, but they are observations rather than automatic explanations. OpenTelemetry also models baggage and is developing profiles; the familiar “three pillars” is therefore a teaching mnemonic, not a closed standard taxonomy.

A diagnosis is an inference supported by correlated evidence and an experiment or change history. A trace can locate a slow span, a metric can establish population impact, and a log can expose event context, yet none of them alone proves root cause. Sampling, missing propagation, aggregation, clocks, and collection outages all constrain what can be concluded.

Модель и предпосылки

• Business traffic remains separate from telemetry export. Loss of a collector should not be presented as proof that the application failed, although synchronous instrumentation can still add cost and must be measured.
• W3C Trace Context propagates identifiers and a sampled flag. It does not promise a complete trace, trustworthy baggage, or causal attribution.
• Dashboards are reached by the investigator through the observability backend; users do not dial Grafana and telemetry does not bypass its collection path.

Проверяемые утверждения

• C1. OpenTelemetry currently documents traces, metrics, logs, and baggage, while profiles are still under development; “three pillars” is not an exhaustive protocol contract.
• C2. Telemetry supports inference but does not itself prove causality or root cause.
• C3. The W3C sampled flag is a tracing decision hint, not a statement that all spans exist or that the trace is unbiased.
• C4. High-cardinality metric labels can make a time-series system operationally expensive; unbounded user or request identifiers belong in exemplars, logs, or traces instead.
• C5. Baggage is propagated application context and can expose sensitive data; it is not automatically stored in spans and must not be trusted as authorization state.
• C6. Cloudflare’s July 2, 2019 outage was caused by a WAF managed-rule regular expression that exhausted CPU, not by BGP.

Исполняемые сценарии

Correlate signals without declaring causality. A user-visible slowdown is bounded by metrics, localized by a trace, and tested against logs or a recent change before a cause is claimed.

Broken context propagation. The business request can succeed while a missing trace header leaves an incomplete trace.

Cardinality and sampling boundary. Bounded metrics describe the population while sampled traces provide examples with an explicit selection policy.

Telemetry pipeline failure. Collector failure produces an observability gap and a separate operational alert; it is not retroactively interpreted as an application outage.

Ошибки проектирования

• Не называйте корреляцию root cause: рост CPU и latency одновременно допускает несколько причин.
• Не переносите user ID, URL или request ID в неограниченный metric label.
• Не считайте отсутствие span доказательством отсутствия вызова: возможны sampling, loss или broken propagation.
• Не соединяйте пользователя напрямую с Grafana и не выдавайте прошлый Cloudflare WAF/CPU outage за BGP-инцидент.

Границы гарантии

• Полнота и стоимость зависят от instrumentation coverage, sampling policy, retention, clocks and collector health.
• Telemetry backend должен иметь собственные SLO, capacity limits и redaction policy.
• Observability улучшает обнаружение и проверку гипотез, но не заменяет safety controls, testing или domain invariants.

Связанные материалы

Официальные и первичные источники

1. OpenTelemetry: Observability primer
2. OpenTelemetry: Signals
3. W3C Trace Context Recommendation
4. Prometheus: Metric and label naming
5. OpenTelemetry: Baggage
6. Cloudflare: Details of the outage on July 2, 2019