System Design Cases
Distributed Tracing
Distributed tracing с OpenTelemetry: W3C trace context propagation, head-based vs tail-based sampling, hot trace investigation, anti-patterns с async boundaries.
Distributed tracing: context, sampling, volume, and inference limits
A distributed trace is a set of recorded spans that share a trace identifier. Parent/child relationships model nested work; span links can express asynchronous, batch, or cross-trace relationships without pretending there is one parent. Instrumentation timestamps and attributes describe recorded work, but a waterfall is not a proof of the unobserved queue, network, scheduler, or root cause.
Head sampling decides before the whole trace is known. If an SDK sampler returns DROP, the collector cannot later recreate missing spans. Tail sampling decides among spans that actually reached a stateful sampling processor, which must route a trace consistently, buffer it, handle late spans, and cap memory. Both methods can bias analysis unless selection probabilities and lost data are understood.
Модель и предпосылки
- W3C
traceparentcarries version, trace-id, parent-id, and flags;tracestatecarries vendor state. The sampled flag is a propagation signal, not a completeness certificate. - Volume is calculated from measured encoded/exported bytes. At 1,000,000 spans/s and 1 KiB/span, raw input is 3.6864 TB/hour before protocol, compression, replication, indexes, or retention.
- OTLP export is a telemetry path separate from the business path; backpressure/drop policy must be bounded so telemetry failure does not create an uncontrolled application outage.
Проверяемые утверждения
- C1. W3C Trace Context standardizes propagation fields and flags but does not guarantee complete, trustworthy, or causally sufficient telemetry.
- C2. Spans support parents, links, events, status, attributes, and kinds; links are useful when one parent does not express an asynchronous relationship.
- C3. A head sampler DROP decision produces a non-recording span, so a downstream tail sampler cannot recover the missing span data.
- C4. Tail sampling is stateful and bounded: consistent trace routing, decision wait, capacity, eviction, and late spans influence completeness.
- C5. Trace storage sizing includes span rate, encoded size, protocol, compression, replication, indexes, and retention rather than one undocumented GB/hour number.
Исполняемые сценарии
Propagate request context. The gateway and services propagate trace context across actual calls while each component creates its own correctly related span.
Head drop cannot be revived. A head decision made before downstream outcomes can discard the recording; tail logic never sees the absent data.
Asynchronous span link. A producer injects context into message metadata and a consumer uses the documented parent/link model across queue delay.
Volume and tail-buffer pressure. Raw sizing uses measured rate and bytes, while the tail processor exposes explicit eviction and late-span loss.
Ошибки проектирования
- Не обещайте, что tail sampler восстановит spans, уже отброшенные head sampler.
- Не считайте sampled trace unbiased population without weights/selection analysis.
- Не передавайте секреты или authorization state в baggage/tracestate.
- Не называйте самый длинный span root cause без проверки queueing, retries, clock and downstream evidence.
Границы гарантии
- Clock skew, async work, batching, retries and broken propagation change waterfall interpretation.
- Collector capacity limits produce explicit observability gaps; fail-open/fail-closed behavior belongs in the telemetry SLO.
- Cost and retention must be measured with real encoded spans, not a universal byte-per-span constant.